# Overmassive black holes at cosmic noon: linking the local and the high-
redshift Universe

Mar Mezcua Institute of Space Sciences (ICE, CSIC), Campus UAB, Carrer de
Magrans, 08193 Barcelona, Spain  Institut d’Estudis Espacials de Catalunya
(IEEC), Edifici RDIT, Campus UPC, 08860 Castelldefels (Barcelona), Spain
Fabio Pacucci Center for Astrophysics $|$ Harvard & Smithsonian, Cambridge, MA
02138, USA  Black Hole Initiative, Harvard University, Cambridge, MA 02138,
USA  Hyewon Suh Gemini Observatory/NSF’s NOIRLab, 670 N. A’ohoku Place, Hilo,
HI 96720, USA  Malgorzata Siudek Institute of Space Sciences (ICE, CSIC),
Campus UAB, Carrer de Magrans, 08193 Barcelona, Spain  Instituto Astrofisica
de Canarias, Av. Via Lactea s/n, E38205 La Laguna, Spain  Priyamvada Natarajan
Department of Astronomy, Yale University, New Haven, CT 06511, USA  Department
of Physics, Yale University, New Haven, CT 06520, USA  Black Hole Initiative
at Harvard University, 20 Garden St., Cambridge MA, 02138, USA

(Received February 28, 2024; Revised April 5, 2024; Accepted April 8, 2024)

###### Abstract

We report for the first time a sample of 12 supermassive black holes (SMBHs)
hosted by low-mass galaxies at cosmic noon, i.e., in a redshift range
consistent with the peak of star formation history: $z\sim 1-3$. These black
holes are two orders of magnitude too massive for the stellar content of their
hosts when compared with the local relation for active galaxies. These
overmassive systems at cosmic noon share similar properties with the high-$z$
sources found ubiquitously in recent James Webb Space Telescope (JWST) surveys
(same range of black hole-to-stellar mass ratio, bolometric luminosity, and
Eddington ratio). We argue that black hole feedback processes, for which there
is possible evidence in five of the sources, and the differing environments in
galactic nuclei at these respective epochs play a key role in these
overmassive systems. These findings contribute to our understanding of the
growth and co-evolution of SMBHs and their host galaxies across cosmic time,
offering a link between the early Universe ($z>4$) observed by JWST and
observations of the present-day Universe ($z\lesssim 1$).

Active galaxies (17) — Dwarf galaxies(416) — Active galaxies(17) —
Supermassive black holes(1663) —Galaxy nuclei(609)

††journal: ApJL

##  1 Introduction

Supermassive black holes (SMBHs; $M_{\mathrm{BH}}>10^{6}$ M⊙) reside at the
center of most massive (with typically a stellar mass $M_{\mathrm{*}}\sim
10^{11}$ M⊙) galaxies (e.g., Kormendy & Ho 2013) and grow through the
accretion of matter and via coalescences during galaxy mergers (Natarajan
2004; Merritt & Milosavljević 2005; Volonteri 2012), which can ignite active
accretion (i.e. active galactic nuclei, AGN).

Tight correlations between BH mass and host galaxy properties (e.g., bulge
mass, stellar velocity dispersion; e.g., Magorrian et al. 1998; Ferrarese &
Merritt 2000; Gebhardt et al. 2000) strongly suggest that the growth of
galaxies is linked to that of their central BHs (e.g., Silk & Rees 1998),
which can be accounted for if a fraction of the AGN energy output couples with
the galactic medium and modulates star formation (thus regulating the growth
of the host galaxy; see Zhuang & Ho 2023 for a recent review). Despite the key
role SMBHs are now believed to play in galaxy formation and evolution,
understanding their origin remains a challenge. The discovery of SMBHs as
heavy as $10^{10}$ M⊙ at redshifts $z\sim$ 6-7 ($\sim$700 Myr after the Big
Bang; see Fan et al. 2023 for a review) and of $10^{6}$-$10^{7}$ M⊙ at even
earlier epochs ($z$ = 8.7, Larson et al. 2023; $z$ = 10.3, Bogdán et al. 2024;
$z$ = 10.6, 440 Myr after the Big Bang, Maiolino et al. 2024) suggests these
SMBHs could have started from seed BHs with masses in the range of $10^{2}$ \-
$10^{5}$ M⊙ (also dubbed intermediate-mass BHs) as early as $z\sim$ 20 (e.g.,
Volonteri 2010; Mezcua 2017; Greene et al. 2020; Pacucci & Loeb 2022). Broadly
speaking, two classes of seeding models - light and heavy - have been proposed
and studied in detail (e.g., Natarajan 2014). Light seed BHs of $\sim$100 M⊙
are expected to have formed from the death of the first generation of
Population III stars; while heavier seed BHs of $\sim 10^{4}-10^{5}$ M⊙ could
have formed via direct collapse of pre-galactic gas (e.g., Loeb & Rasio 1994;
Lodato & Natarajan 2006). Other possibilities include early super-Eddington
growth (Volonteri & Rees 2005), stellar mergers in early nuclear stellar
clusters (e.g., Portegies Zwart et al. 1999; Devecchi & Volonteri 2009), rapid
growth of light seeds via wind-fed accretion in early nuclear star clusters
(Alexander & Natarajan 2014), and primordial BHs (e.g., Cappelluti et al.
2022; Ziegler & Freese 2022). There is growing evidence that the recent
detection of $10^{6}$-$10^{7}$ M⊙ BHs at $z>$ 8 favors the heavy seeding
scenario, as lighter seeds would require periods of super-Eddington accretion
(e.g., Pacucci & Loeb 2022; Bogdán et al. 2024; Larson et al. 2023; Maiolino
et al. 2024; Natarajan et al. 2024).

At low redshifts, insights into seeding can be obtained from a multiplicity of
observational probes, ranging from the low-mass end of the BH occupation
fraction (e.g., Ricarte & Natarajan 2018; Burke et al. 2023; Beckmann et al.
2023) to the BH mass function (e.g., Gallo & Sesana 2019; Habouzit et al.
2021; Sicilia et al. 2022) and luminosity function (e.g, Volonteri et al.
2008; Beckmann et al. 2023). These studies triggered a quest for actively
accreting intermediate-mass BHs in dwarf galaxies, where they can be detected
in large numbers as low-mass ($M_{\mathrm{BH}}\lesssim 10^{6}$ M⊙) AGN (e.g.,
see review by Reines 2022). The use of optical spectroscopy has thus far
yielded detections of hundreds of AGN in dwarf galaxies and provided estimates
of the AGN occupation fractions (taken as a proxy for BH occupation fraction)
ranging from $<1$ % (Reines et al. 2013; Salehirad et al. 2022) to $\sim 20$ %
(Polimera et al. 2022; Mezcua & Domínguez Sánchez 2020, 2024). AGN variability
studies of large samples of nearby sources are also providing interesting
constraints on the local BH occupation fractions (Burke et al. 2024). AGN
fractions of $<$1% corrected for completeness have been derived from X-ray
surveys (e.g., Mezcua et al. 2018a; Birchall et al. 2020, 2022; Bykov et al.
2024). Meanwhile, theoretical predictions for the AGN fraction in dwarf
galaxies have also been derived (e.g, Pacucci et al. 2021). While these
estimated AGN fractions seem to favor heavy seeding BH models, a direct link
between the early seeds and the BHs powering AGN in low-redshift dwarf
galaxies remains to be firmly established.

Recently, Siudek et al. (2023) identified a sample of 4,315 AGN in dwarf
galaxies at 0.5 $<z<$ 0.9, the largest of its kind. The average BH mass of
this sample is log $M_{\mathrm{BH}}$ = 8.2 M⊙, derived from correlations
between narrow emission lines (Baron & Ménard 2019). This average BH mass is
two orders of magnitude more massive than expected from local BH-galaxy
scaling relations. Based on broad emission lines, Mezcua et al. (2023)
recently identified another sample of overmassive BHs in dwarf galaxies at
$z\sim$0.4-0.9. Interestingly, the James Webb Space Telescope (JWST) in its
first year of operation has found tens of such overmassive BHs at $z=4-10$,
which lie $10-100$ times above the local $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$
relation (Pacucci et al. 2023). Many of these overmassive SMBHs are hosted in
low-mass galaxies of $M_{\mathrm{*}}\sim 10^{8}-10^{10}\,M_{\odot}$ (e.g.,
Bogdán et al. 2024; Furtak et al. 2024; Harikane et al. 2023; Kocevski et al.
2023; Kokorev et al. 2023; Maiolino et al. 2023, 2024; Stone et al. 2023;
Übler et al. 2023).

The finding that these BHs are overmassive with respect to the local
$M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ relation is not a consequence of survey
selection effects, but is rather expected from some theoretical simulations
(e.g., Agarwal et al. 2013; Inayoshi et al. 2022; Trinca et al. 2022),
especially if the SMBHs arise from heavy seeds (e.g., Scoggins et al. 2023).
Recent results from the ASTRID, Illustris TNG50 simulation suites also predict
the existence of such an overmassive SMBH population in the local Universe
(Weller et al. 2023). At these late epochs, environmental effects like tidal
stripping of the stellar component of host galaxies are implicated in causing
the central SMBHs in dwarf galaxies to be overmassive compared to observed
scaling relations (e.g., Ferré-Mateu et al. 2018, 2021).

In this Letter we report on a sample of 12 low-mass galaxies hosting
overmassive BHs at $z\sim 1-3$, constituting the first such sample at the peak
of cosmic star formation history (referred to as the ‘cosmic noon’; Madau &
Dickinson 2014). These $z\sim 1-3$ overmassive BHs and the $z\sim 0.4-0.9$
from Mezcua et al. (2023) share the same properties as the $z>4$ JWST
overmassive BHs, allowing us to study for the first time low-mass galaxies in
the high-redshift Universe with lower redshift counterparts and to probe BH-
galaxy co-evolution across cosmic time. The sample and data analysis are
described in Sect. 2. The results obtained are reported in Sect. 3. Discussion
and conclusions are provided in Sect. 4. We adopt a $\Lambda$-CDM cosmology
with $H_{0}=70$ km s-1 Mpc-1, $\Omega_{\Lambda}=0.73$ and $\Omega_{m}=0.27$.

##  2 Sample and Analysis

The sample selection is performed following the same procedure as in Mezcua et
al. (2023), which we summarize here.

###  2.1 Stellar mass measurements

We start from a parent sample of 1,161 broad-line AGN galaxies identified in
the VIMOS Public Extragalactic Redshift Survey (VIPERS), which includes
$\sim$90,000 spectra ranging from $z$ = 0.1 to $z$ = 4.56 (Scodeggio et al.
2018). We fit the multiwavelength spectral energy distribution (SED) of these
1,161 galaxies from ultraviolet to infrared wavelengths using a modified
version of the SED code of Suh et al. (2019). This includes a combination of
galaxy and AGN templates, using the same SED libraries as in AGNfitter
(Calistro Rivera et al., 2016). The best fit is initially determined using a
$\chi^{2}$ minimization. We note that the delineation of the flux contribution
from the stellar component versus the AGN is extremely challenging and here we
do it adopting multiple independent empirical methods.

For those sources with an X-ray detection in the XMM-XXL catalogue (Chiappetti
et al. 2018; 452 out of the 1,161 broad-line AGN), we confirm that the rest-
frame luminosity at 2500Å of the best-fitting AGN component correlates with
the X-ray luminosity, in agreement with the X-ray-to-ultraviolet correlation
of AGN (e.g., Lusso & Risaliti 2016).

To account for the degeneracies inherent to the SED fitting, we derive a
probability distribution function (PDF) for the stellar mass that considers an
AGN fraction (fAGN) ranging from 0 to 1. We then perform the SED fitting
assuming that the galaxy emission dominates over the AGN in the K-band, using
the oldest stellar population possible in order to obtain a conservative upper
limit on the best-fit stellar mass. The difference between the highest most
probable value (MsPDF) and the best-fit stellar mass (MsBEST) is then taken as
the uncertainty in the stellar mass (see Fig. 4, top right panel). An
additional 0.2 dex is added to the uncertainties in order to account for
differences in the stellar population models arising from factors such as the
choice of the Initial Mass Function. The star formation rate (SFR) is derived
from the best-fit SED as SFR $\propto e^{t/\tau}$, where the characteristic
time of the exponentially decaying star formation histories of the stellar
population models range from $\tau$=0.1 to 30 Gyr.

Since SED-fitted stellar masses are typically highly dependent on the SED
fitting code, we also derive the stellar masses independently using the Code
Investigating GALaxy Emission (CIGALE; Boquien et al. 2019) and adopting
different parameters for the models of the stellar populations and star
formation history, dust emission and attenuation, and AGN emission. For those
sources with X-ray emission, we also perform the SED fitting with the X-CIGALE
code (Yang et al. 2020), which incorporates X-ray fluxes. We find that even
with these different parameterizations the stellar masses are consistent with
those derived from AGNfitter. More details on the SED fitting can be found in
the Appendices A and B in Mezcua et al. (2023).

The sample of low-mass galaxies reported here are selected as having $z>1$
(those sources with $z<1$ are reported in Mezcua et al. 2023) and an MsBEST of
log $M_{\mathrm{*}}\leq$ 9.5 M⊙, which is the typical threshold considered in
studies of AGN in dwarf galaxies. This mass range corresponds approximately to
the stellar mass of the Large Magellanic Cloud (e.g., Reines et al. 2013,
2020; Mezcua et al. 2016, 2018a, 2019; Mezcua & Domínguez Sánchez 2020, 2024).
We consider only those galaxies with an upper MsPDF not extending beyond 1010
M⊙ to restrict our sample to the low-mass regime. We note that, because of the
large uncertainties in the stellar masses at intermediate to high redshifts,
it is more appropriate to use the term ‘low-mass’ rather than ‘dwarf’ to refer
to those galaxies with $M_{\mathrm{*}}<10^{10}$ M⊙ found at $z\gtrsim 1$, as
at these redshifts we can only confidently distinguish between low-mass and
massive galaxies. This nomenclature we argue applies not only to this Letter
but also to those samples of low-mass galaxies derived using JWST data. From
now on, the term ‘dwarf’ will therefore be used to refer only to local
sources.

###  2.2 Emission line fitting

The cuts in redshift and stellar mass applied above yield a sample of 13 low-
mass AGN galaxies at $z>1$. To confirm the presence of broad emission lines in
these sources, we use the public Python QSO fitting code (PyQSOFit; Guo et al.
2018) to fit their optical spectrum. PyQSOFit fits the continuum emission with
a power-law using a few emission-line-free regions and then subtracting it.
The rest-frame MgII and CIV emission lines are then fitted using typically two
components, one broad and one narrow, where the narrow one is defined as
having a Full Width at Half Maximum (FWHM) $<$ 1,200 km s-1 (Shen et al.
2019). As a result, PyQSOFit outputs the line flux, FWHM, equivalent width and
dispersion of the broad and narrow components as well as the continuum
luminosity at $1350\rm\AA$ and $3000\rm\AA$ (when available). A Monte Carlo
approach is used to compute the uncertainties in the emission line
measurements.

For five of the 13 low-mass AGN galaxies, SDSS111Sloan Digital Sky Survey,
https://www.sdss.org DR14 spectra are also available. We fit these SDSS
spectra using the same procedure as described above. We favor the SDSS fit
over that of VIPERS for three sources, for which the VIPERS spectrum contains
only one emission line (e.g., only the CIV line is in the VIPERS spectrum
while in the SDSS there are additional lines, namely, CIV and Ly$\alpha$) and
the $\chi^{2}$ of the SDSS fit is better than the one derived from VIPERS. For
one of these three sources, 401126746, the CIV line of the SDSS spectrum shows
an artifact that prevents a reliabe use of the FWHM to derive the BH mass. For
this source, the BH mass is derived from the FWHM and luminosity of the
H$\alpha$ emission line (Suh et al. 2020) detected in Gemini/GNIRS Fast
Turnaround observations (PI: Suh; see Appendix A).

Five of the sources show possible asymmetric components in the Ly$\alpha$, CIV
or CIII] emission suggestive of outflows (see e.g., Fig. 4). A thorough study
of AGN outflows is out of the scope of this paper. However, in order to avoid
any biases in the BH mass estimations, for these galaxies the fit of the CIV
emission line is attempted with one additional broad component in order to
take the possible outflow into account. For four out of the five sources, this
does not improve the fit. For the remaining source, 127008752, the addition of
a broad component yields a slight improvement of the fit ($\chi^{2}$ = 0.9
versus $\chi^{2}$ = 1.4 when only using one broad component) but also
increases the FWHM of the broad component used to derive the BH mass by a
factor 1.2. To be as conservative as possible, we thus proceed with the fit
that provides the lowest value of FWHM.

In total, we find that the optical VIPERS, SDSS, or near-infrared Gemini/GNIRS
spectrum are reliably fitted by PyQSOFit for 12 out of the 13 low-mass AGN
galaxies. Our final sample is thus composed of 12 AGN low-mass galaxies at
$z>$1\. The spectral fit of one of these 12 sources is shown in the Appendix,
Fig. 4, middle panel.

###  2.3 Black hole masses

BH masses are derived from the width of the MgII (if $z<2$) or CIV or
H$\alpha$ (if $z>2$) broad emission line components and the line luminosity or
adjacent continuum at 1350 Å or 3000 Å, when available. We use the single-
epoch virial calibrations from Vestergaard & Peterson (2006) and Shen & Liu
(2012) based on a mean virial factor $\epsilon\sim$1 (e.g., Onken et al. 2004;
Grier et al. 2013) and with a scatter of $\sim$0.3 dex (e.g., Shen & Liu
2012). Adding in quadrature the measurement uncertainties of $\sim$0.1 dex
results in a total BH mass uncertainty of $\sim$0.4 dex. A word of caution is
warranted regarding this methodology for estimating the BH mass for high-$z$
objects. Single-epoch virial calibrations, such as the one from Vestergaard &
Peterson (2006), are calibrated in the local Universe, typically at $z\ll 1$.
Hence, while this constitutes the only viable methodology available with
current data, additional systematic effects could be present in the BH mass
estimates. This should however not threaten the results here reported, which
for most sources should hold unless the BH masses are overestimated by a
factor $\sim 60$ (Pacucci et al. 2023).

AGN bolometric luminosities are derived from the continuum luminosities at
$1350\rm\AA$ and $3000\rm\AA$ applying the bolometric correction factors of
Richards et al. (2006). Eddington rates are then derived using
$\lambda_{\mathrm{Edd}}$ = $L_{\mathrm{bol}}$/($M_{\mathrm{BH}}\times
1.3\times 10^{38}$).

##  3 Results

The new sample of 12 low-mass AGN galaxies here reported have spectroscopic
redshifts ranging from $z$ = 1.32 to 2.78, constituting the first such sample
at the key epoch of cosmic noon ($z\sim$ 1-3) where cosmic star formation
activity and BH growth reached their peaks (e.g., Madau & Dickinson 2014). The
SFRs of this ‘cosmic noon’ sample of 12 low-mass galaxies are in the range log
SFR = 1.1 to 1.7 M⊙ yr-1, which locates them above the main-sequence of star-
forming galaxies (e.g., Whitaker et al. 2014). Five of the sources have a
0.2-12 keV X-ray counterpart in the 4XMM-DR13 catalogue (Webb et al. 2020),
with a k-corrected luminosity ranging log L0.2-12keV = 44.2 - 44.8 erg s-1
that confirms their AGN nature. We note that, given the estimated bolometric
luminosity of these sources (i.e., log L${}_{\mathrm{bol}}\sim$ 45-46 erg s-1,
see Table 1), this leads to an average hard X-ray bolometric correction of
$k_{\mathrm{X}}\sim 20$, which is in perfect agreement with the universal
bolometric corrections derived in e.g., Duras et al. (2020). The most relevant
information about each low-mass AGN galaxy is provided in Table 1.

Table 1: Properties of the ‘cosmic noon’ sample of 12 low-mass galaxies at
$z\sim 1-3$ hosting overmassive BHs.

VIPERS | R.A. | Dec. | $z$ | log $M_{\mathrm{*}}$ | log $M_{\mathrm{BH}}$ | log $L_{\mathrm{bol}}$ | Survey | Broad | log $L_{\mathrm{X}}$
---|---|---|---|---|---|---|---|---|---
ID | (J2000) | (J2000) |  | (M⊙) | (M⊙) | (erg s-1) |  | Line | (erg s-1)
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10)
402088938 | 22:07:12.95 | +01:11:15.6 | 1.322 | 9.1${}^{+0.5}_{-0.1}$ | 8.6$\pm$0.3 | 45.0 | VIPERS | MgII | –
103212570 | 02:10:19.15 | -05:38:55.4 | 1.384 | 9.4${}^{+0.5}_{-0.2}$ | 8.9$\pm$0.3 | 45.2 | VIPERS | MgII | –
116100529 | 02:26:07.33 | -05:06:44.2 | 1.628 | 9.4${}^{+0.9}_{-0.1}$ | 7.3$\pm$1.2 | 45.2 | VIPERS | MgII | 44.2$\pm$0.3
117162807 | 02:28:22.40 | -04:49:59.1 | 1.628 | 9.0${}^{+0.7}_{-0.0}$ | 8.4$\pm$0.3 | 45.4 | VIPERS | MgII | 44.5$\pm$1.6
113197834 | 02:13:08.20 | -04:43:06.5 | 2.565 | 9.5${}^{+0.8}_{-0.1}$ | 8.1$\pm$0.4 | 46.1 | SDSS | CIV | 44.7$\pm$0.3
401126746 | 22:02:56.52 | +01:21:56.5 | 2.587 | 9.2${}^{+0.7}_{-0.1}$ | 8.8$\pm$0.3 | 45.3 | SDSS/Gemini | CIV/H$\alpha$ | –
405118143 | 22:17:53.28 | +01:22:02.8 | 2.614 | 9.5${}^{+0.6}_{-0.1}$ | 7.7$\pm$0.3 | – | VIPERS | CIV | –
126027215 | 02:28:46.40 | -04:32:34.6 | 2.696 | 9.2${}^{+1.0}_{-0.2}$ | 7.9$\pm$0.3 | – | VIPERS | CIV | 44.8$\pm$0.2
112102440 | 02:10:15.93 | -05:08:32.8 | 2.738 | 9.0${}^{+1.1}_{-0.2}$ | 8.4$\pm$0.3 | – | VIPERS | CIV | 44.7$\pm$0.2
411013198 | 22:15:28.50 | +01:48:15.1 | 2.744 | 9.5${}^{+0.6}_{-0.1}$ | 8.3$\pm$0.4 | 45.9 | SDSS | CIV | –
111011166 | 02:07:37.71 | -05:34:31.7 | 2.744 | 9.4${}^{+0.8}_{-0.1}$ | 8.0$\pm$0.3 | – | VIPERS | CIV | –
127008752 | 02:32:35.83 | -04:39:16.2 | 2.768 | 9.2${}^{+0.8}_{-0.1}$ | 8.3$\pm$0.3 | – | VIPERS | CIV | –

Column designation: (1) VIPERS ID; (2,3) right ascension and declination; (4)
redshift; (5) stellar mass derived from SED fitting; (6) BH mass derived from
single-epoch virial calibrations; (7) bolometric luminosity derived from the
monochromatic continuum luminosity at 1350 Å or 3000 Å; (8) survey for
spectroscopy; (9) fitted broad emission line; (10) 0.2-12 keV X-ray
luminosity, if available. The uncertainties in the stellar mass include a 0.2
dex to account for differences in the stellar population models. The
uncertainties in BH mass are the quadratic sum of the measurement
uncertainties ($\sim$0.1 dex) and the systematic uncertainties carried by
single-epoch virial calibrations ($\sim$0.3 dex).

###  3.1 AGN properties

The ‘cosmic noon’ sample of 12 low-mass AGN galaxies at $z\sim 1-3$ have log
$M_{\mathrm{BH}}=7.3-8.9$ M⊙ with an average uncertainty of 0.4 dex. These
SMBH masses are two to three orders of magnitude higher than those of the low-
mass AGN found in local dwarf galaxies ($M_{\mathrm{BH}}\lesssim 10^{6}$ M⊙;
e.g., Reines et al. 2013; Mezcua & Domínguez Sánchez 2020, 2024). The
bolometric luminosities of the ‘cosmic noon’ sample range from log Lbol = 44.9
to 46.1 erg s-1 and are again orders of magnitude higher than those of AGN
dwarf galaxies in the local Universe (log L${}_{\mathrm{bol}}\sim$ 40-42 erg
s-1; e.g., Mezcua & Domínguez Sánchez 2020, 2024). The Eddington ratios are in
the range $\lambda_{\mathrm{Edd}}$ = 0.02 - 0.8, with a median value
$\lambda_{\mathrm{Edd}}$ = 0.2, indicating that the AGN low-mass galaxies at
$z\sim 1-3$ are mostly accreting at sub-Eddington rates.

The BH masses, bolometric luminosities, and Eddington ratios of the low-mass
AGN galaxies at $z\sim 1-3$ are very similar to those of the seven AGN dwarf
galaxies found by Mezcua et al. (2023) at $z=0.35-0.93$ (log $M_{\mathrm{BH}}$
= 7.6-8.7 M⊙, log Lbol = 44.8 to 45.4 erg s-1, median $\lambda_{\mathrm{Edd}}$
= 0.1). The galaxy properties of both samples are also very similar (same
range of stellar masses and of SFR), with all the sources being star-forming
galaxies (see Mezcua et al. 2023). All together indicates that the ‘cosmic
noon’ AGN low-mass galaxies and those at $z<1$ are similar sources just
observed at different cosmic epochs.

###  3.2 The $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ scaling relation

The AGN dwarf galaxies at $z\sim 0.4-0.9$ of Mezcua et al. (2023) were found
to host BHs more massive than expected from the local
$M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ scaling relation of AGN (e.g., at $z<0.05$,
Reines & Volonteri 2015; from now on RV2015) and that at $z\sim 0.4-2.5$ of
Suh et al. (2020) (from now on Suh+2020). To investigate further the ‘cosmic
noon’ sample of 12 AGN low-mass galaxies at $z\sim 1-3$ here reported, we
locate them in the $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ diagram and compare them
to the RV2015 sample of AGN in dwarf and massive galaxies, the Suh+2020 sample
of AGN in massive galaxies, and the Shankar et al. (2016) local correlation
for inactive galaxies corrected for resolution-related effects (see Fig. 1).
As for the Mezcua et al. (2023) sources, the new sample at $z\sim 1-3$ is also
found to be overmassive with respect to the stellar mass according to the
local scaling relation for AGN. The BH mass offset ($\Delta M_{\mathrm{BH}}$)
from the $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ can be derived using a Monte Carlo
approach, by assigning 100 random variables to the $M_{\mathrm{BH}}$ and
$M_{\mathrm{*}}$ distributions of each source and calculating $\Delta
M_{\mathrm{BH}}$ based on the distribution of 1002 possibilities over the
number of sources (e.g., Mezcua et al. 2018b; see Appendix D in Mezcua et al.
2023). For the seven overmassive BHs in Mezcua et al. (2023) at $z=0.35-0.93$,
the median of the BH mass offset from the local + intermediate-z
$M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ correlation of Suh+2020 was $\Delta
M_{\mathrm{BH}}$ = 3.2 $\pm$ 1.3 with a significance of 100% (3$\sigma$
level). The same offset and significance are also now obtained for the new
sample of 12 AGN in low-mass galaxies at $z\sim 1-3$, indicating that these
sources at cosmic noon are also overmassive. Combining the seven $z<1$ sources
of Mezcua et al. (2023) with the new ‘cosmic noon’ sample of 12 sources, we
also find the same $\Delta M_{\mathrm{BH}}$ but at a 5$\sigma$ level,
suggesting once more that the in total 19 AGN in low-mass galaxies found from
$z=0.35$ to $z=2.7$ in the VIPERS survey share the same properties. When
considering the local relation of inactive galaxies corrected for resolution-
related selection effects of Shankar et al. (2016), the sources are still as
offset as they are from the local scaling relation for AGN; this is a result
that will be further discussed in Sect. 4.

Figure 1: $M_{\mathrm{BH}}$ versus $M_{\mathrm{*}}$ for the VIPERS sample of
AGN dwarf galaxies at $z<1$ of Mezcua et al. (2023) (blue squares), the new
‘cosmic noon’ sample of AGN in low-mass galaxies here reported (red squares),
and a compilation of JWST AGN in low-mass galaxies at $z>4$ (purple squares).
We also show for comparison the low-z AGN of Reines & Volonteri (2015)
(RV2015, blue dots) and the intermedidate-z AGN of Suh et al. (2020)
(Suh+2020, red dots), whose masses have been computed using the same procedure
and parameters as the VIPERS samples. The solid black line shows the local +
intermediate-z $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ correlation found for the
combination of the RV2015 and Suh+2020 samples with a 1$\sigma$ scatter of 0.5
dex. The red dashed line shows the high-z $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$
correlation found by Pacucci et al. (2023) for AGN found at $z>4$ by JWST
based on the detection of broad H$\alpha$ or H$\beta$ emission. The green
dashed-dotted line shows the Shankar et al. (2016) local correlation for
inactive galaxies.

###  3.3 Overmassive BHs across cosmic time

Tens of overmassive BHs have been recently found by the JWST at $z>4$. We
compile here all those AGN found by the JWST based on the detection of broad
H$\alpha$ or H$\beta$ emission in low-mass galaxies of log
$M_{\mathrm{*}}\leq$ 10 M⊙ at $z\sim 4-8$, shown as ‘JWST’ in Figs. 1-3:
Harikane et al. (2023), 10 sources; Übler et al. (2023), one source; Maiolino
et al. (2023), 12 sources; Stone et al. (2023), one source; Furtak et al.
(2024), one source; Kokorev et al. (2023), one source; Yue et al. (2023), two
sources. In the case of H$\alpha$ emission (Harikane et al. 2023; Übler et al.
2023; Maiolino et al. 2023), the BH masses have been computed using the virial
correlations of Reines et al. (2013) (see Pacucci et al. 2023). For the
Harikane et al. (2023) sources, we have added in quadrature the typical 0.3
dex scatter of the virial relations to the instrumental BH mass error. In the
case of H$\beta$ emission (Stone et al. 2023; Furtak et al. 2024; Kokorev et
al. 2023; Yue et al. 2023), the BH masses have been derived as for the VIPERS
sources, using the relations from Vestergaard & Peterson (2006) and Shen & Liu
(2012), or from calibrations of Greene & Ho (2005). As for the VIPERS sources,
the stellar masses of half of the JWST sources (Übler et al. 2023; Maiolino et
al. 2023; Kokorev et al. 2023) have been derived via SED fitting, allowing us
only to distinguish between low-mass and massive galaxies. For the remaining
JWST sources (Harikane et al. 2023; Stone et al. 2023; Yue et al. 2023), the
stellar masses have been more reliably derived using spatial AGN-host
decomposition.

The total of 28 JWST AGN in low-mass galaxies at $z>4$ here compiled are
offset from the local $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ correlation of active
galaxies by $\Delta M_{\mathrm{BH}}$ = 2.8 $\pm$ 1.9 at a 5$\sigma$ level,
confirming that they are overmassive. Both these JWST sources and the VIPERS
ones at $z=0.35-2.7$ are indeed found to sit on the
$M_{\mathrm{BH}}$-$M_{\mathrm{*}}$ at $z=4-7$ derived by Pacucci et al. (2023,
red line in Fig. 1), which deviates at more than 3$\sigma$ confidence level
from the local relation for AGN. We have run the same algorithm used to infer
the $z>4$ relation in Pacucci et al. (2023), adding the 19 low-mass galaxies
investigated here at $z<3$. We find the following values for the intercept $b$
and the slope $m$ of the linear relation: $b=-2.27\pm 0.67$, and $m=1.10\pm
0.07$. Remarkably, these values agree with the ones initially inferred in
Pacucci et al. (2023): $b=-2.43\pm 0.83$, and $m=1.06\pm 0.09$. From a
statistical perspective, this test suggests that the $z>4$ sample discovered
by JWST and the VIPERS sample at $z<3$ (including the new sources here
reported and those from Mezcua et al. 2023) belong to the same population.

The VIPERS sources at $z<1$ of Mezcua et al. (2023), the new VIPERS sample at
$z\sim 1-3$, and the JWST sources at $z>4$ also share the same range of
$M_{\mathrm{BH}}$/$M_{\mathrm{*}}$ ratios (see Fig. 2), ranging from
$\sim$0.2% to $\sim$38%. Most of the values are higher than the 1% threshold
used to define outlier sources (together with the $M_{\mathrm{BH}}>10^{7}$ M⊙
criterion, van Son et al. 2019, which all of the VIPERS sources and most of
the JWST here compiled fullfill). Therefore, the AGN in low-mass galaxies at
$z\sim 0.4-8$ considered here, both from the VIPERS and JWST surveys, host BHs
that are outliers with regard to the local $M_{\mathrm{BH}}$-$M_{\mathrm{*}}$
relation of active galaxies. This is independent of whether the
$M_{\mathrm{BH}}$/$M_{\mathrm{*}}$ ratio evolves with redshift (Decarli et al.
2010; Merloni et al. 2010; Trakhtenbrot & Netzer 2010; Bennert et al. 2011;
Ding et al. 2020; Caplar et al. 2018; Pacucci & Loeb 2024) or not (Jahnke et
al. 2009; Cisternas et al. 2011; Mullaney et al. 2012; Sun et al. 2015;
Setoguchi et al. 2021), see Fig. 2.

Figure 2: Ratio of $M_{\mathrm{BH}}$/$M_{\mathrm{*}}$ versus redshift for the
VIPERS sample of AGN dwarf galaxies at $z<1$ of Mezcua et al. (2023), the new
VIPERS ‘cosmic noon’ sample of AGN in low-mass galaxies here reported, and a
compilation of JWST AGN in low-mass galaxies at $z>4$. We show for comparison
the non-evolution found by Suh et al. (2020) for $z\sim 0-2.5$ (black line,
1$\sigma$ scatter$\sim$0.5 dex), Sun et al. (2015) for $z\sim 0-2$ (brown
line), and Setoguchi et al. (2021) for $z\sim 1.2-1.7$ (red line), and the
z-evolution found by Decarli et al. (2010) for $z\sim 0-3$ (dashed blue line),
Trakhtenbrot & Netzer (2010) for $z\sim 0.1-2$ (dashed yellow line), Bennert
et al. (2011) for $z\sim 1-2$ (dashed green line, including the data of
Merloni et al. 2010), and Pacucci & Loeb (2024) for the JWST $z>4$ population
of overmassive BHs (dashed purple line). We note that the Suh+2020 sample is
the only one for which the BH and stellar masses have been computed using the
same procedure and parameters as the VIPERS samples.

In addition to being overmassive, the JWST AGN in low-mass galaxies at $z>4$
also share a similar distribution of bolometric luminosities as the VIPERS
sources at $z<1$ and the new VIPERS sample at $z\sim 1-3$ (see Fig. 3, top
panel), with log Lbol = 43.7 to 47.2 erg s-1, and similar distribution of
Eddington ratios (see Fig. 3, bottom panel), with most of the sources
accreting at sub-Eddington rates (ie., $\lambda_{\mathrm{Edd}}<$ 1). Applying
a Mann-Whitney statistical test between the VIPERS sample (at $z\sim 0.4-3$)
and the JWST sample (at $z>4$) returns a p-value greater than 0.02 both for
the Eddington ratio and the bolometric luminosity distributions, hence we
cannot reject the null hypothesis that the two samples are drawn from the same
distribution at a 98% confidence level. We note though that performing a
robust statistical test to compare two independent samples when the sample
size is very small is challenging. Yet, the statistical results seem to be
consistent with the visual inspection of the histograms presented in Fig. 3,
where we see that the central tendencies, spread, and shapes between the
samples are similar. All these low-mass galaxies hosting overmassive AGN could
thus be similar sources simply detected at different cosmic epochs.

Figure 3: Distribution of bolometric luminosity (top) and Eddington ratio
(bottom) for the sample of AGN dwarf galaxies at $z<1$ of Mezcua et al.
(2023), the new ‘cosmic noon’ sample of AGN in low-mass galaxies here
reported, and a compilation of JWST AGN in low-mass galaxies at $z>4$.

##  4 Discussion and Conclusions

In this Letter, we report on the first sample of significantly overmassive
SMBHs detected at cosmic noon, i.e., in the redshift range $z=1-3$.
Remarkably, these overmassive systems share several similar physical
properties (BH mass, stellar mass, bolometric luminosity, Eddington ratio)
with their high-$z$ counterparts, found ubiquitously in recent JWST surveys.

Eilers et al. (2024) recently investigated four JWST fields at $z>$ 6 and
found that the duty cycle for UV-luminous quasars at these cosmic epochs is
significantly less than unity. This suggests that such high-z SMBHs may
undergo episodic and highly dust-obscured phases of radiatively inefficient
super-Eddington accretion. In our ‘cosmic noon’ sample of AGN low-mass
galaxies as well as that at $z<1$, all the sources are characterized by sub-
Eddington luminosities; most have an Eddington ratio lower than 60% Eddington.
Our sources are all observed at $z<3$ and are intrinsically less likely to
undergo extreme super-Eddington phases, as the availability of cold gas
necessary to fuel such extreme growth phases is much lower at these redshifts
than at $z>6$ (see, e.g., Power et al. 2010). Furthermore, we point out that
Juodžbalis et al. (2024) recently presented the discovery at $z>$ 6 of an
extremely overmassive BH, with a mass of $\sim 40\%$ the total stellar mass of
the host galaxy, which is accreting at the meager rate of $2\%$ Eddington.
Hence, extraordinarily overmassive yet low-luminosity SMBHs have already been
detected in the early Universe.

In a recent study of variability-selected AGN in low-mass galaxies at $z\sim
0.5-4$ drawn from the COSMOS survey (Scoville et al. 2007) and followed up in
the Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP; Aihara et al. 2022) -
a sample coincident in redshift range with the ‘cosmic noon’ sample studied
here - Burke et al. (2024) find that these sources are also overmassive
compared to the local $M_{\rm BH}-M_{*}$ relation for active galaxies.
However, they appear to be consistent with the local relation for inactive
early-type galaxies, as would be the case for some of the overmassive BHs here
presented (at cosmic noon and at $z>4$ with JWST). Local inactive galaxies
with dynamical BH mass measurements tend however to be biased by angular
resolution-related selection effects (e.g., Bernardi et al. 2007; Shankar et
al. 2016). Using the resolution-corrected or debiased correlation of local
quiescent galaxies of Shankar et al. (2016; see also Shankar et al. 2019, eq.
5) we find that both the AGN in low-mass galaxies at cosmic noon and at $z<1$
as well as the JWST ones at $z>4$ are clearly offset from the correlation (see
Fig. 1). Such offset seems to be softened for the JWST sources at $z>4$ when
using the $M_{\rm BH}-\sigma$ correlation, where $\sigma$ is the stellar
velocity dispersion derived from the width of the [OIII] emission line (e.g.,
Juodžbalis et al. 2024). The absence of such line in the VIPERS spectra of the
$z\sim 1-3$ sources prevents probing this for the ‘cosmic noon’ sample.

With $\sim 3$ Gyr of cosmic time elapsed between the median redshift of our
‘cosmic noon’ sample and the JWST’s high-$z$ sample, it is challenging to
connect these samples in a straightforward fashion as the causes for departure
from the local scaling relation are likely to be different at these cosmic
epochs. Nonetheless, some considerations are worth mentioning.

At high redshifts, the existence of overmassive SMBHs can be attributed to
seeding physics. Recent cosmological simulations (e.g., Scoggins et al. 2023)
suggest that heavy seeding, for example as a result of the direct collapse BH
seed formation scenario, will result in an early (i.e., $z>15-20$) ratio of BH
to stellar mass close to unity (Agarwal et al. 2013). In other works (e.g.,
Bhowmick et al. 2024) lower mass seeds of $\leq 10^{3}$ M⊙ are already able to
predict BH masses $\sim$10-100 times higher than expected from local scaling
relations. By cosmic noon, however, we expect two additional important effects
to modulate the scaling relations between central SMBHs and the stellar
content of their host galaxies: (i) feedback processes and (iii) impact of
interactions/mergers leading to tidal stripping.

First, we note an important caveat: selection effects might well be operating
here. For instance, we do know that the highest redshift ($z>6$) JWST detected
sources are likely the most luminous population and are therefore outliers in
terms of their luminosities and inferred BH masses. It is conceivable that
these extremely bright sources are preferentially overmassive. It has been
argued persuasively at least in the case of the $z=10.1$ source UHZ1 that this
system was likely seeded with a massive seed of $10^{4}-10^{5}$ M⊙ causing it
to be overmassive (Bogdán et al. 2024; Natarajan et al. 2024). So heavy
seeding coupled with feedback physics as noted by Pacucci & Loeb (2024) might
be implicated for these extremely high-redshift sources. However, the fact
that we still see overmassive systems $\sim 3$ Gyr after in cosmic time could
suggest that there exists a galaxy population that even by these late times
was not able to build up enough stellar mass to shift towards the local
$M_{\mathrm{BH}}-M_{\star}$ relation.

The finding that these systems are still overmassive at cosmic noon suggests
that BH feedback processes are playing a significant role in shaping their
host galaxies (Koudmani et al. 2022), likely quenching star formation and
stellar growth (e.g, Pacucci & Loeb 2024; Silk et al. 2024). This could
explain the compactness and dust-reddened emission of most of the high-$z$
JWST sources (Greene et al. 2024). Indeed, the presence of AGN outflows is
prominent in one of the most distant AGN (GN-z11; Maiolino et al. 2024) and in
some of the Harikane et al. (2023) sources here considered (for which Harikane
et al. 2023 correct the H$\alpha$ line emission used to derive the BH mass).

In the ‘cosmic noon’ sample here reported, five of the sources show possible
asymmetric components in the Ly$\alpha$, CIV or CIII] emission indicative of
outflows. We note that in the local Universe evidence is growing that AGN
feedback can be equally or even more important than supernova feedback in
shaping dwarf galaxies (Manzano-King et al. 2019; Mezcua et al. 2019; Liu et
al. 2020; Davis et al. 2022; Schutte & Reines 2022), and AGN feedback is
expected to have impacted BH growth in dwarf galaxies across cosmic time
(Mezcua 2019).

A more comprehensive study of the detailed physics of AGN feedback processes
in the overmassive systems at cosmic noon and beyond is required to better
understand the co-evolution and the mass assembly history of stars and the
SMBHs hosted in galaxies. In particular, the role of outflows in modulating
asynchronous BH-galaxy growth and/or growth in tandem is urgently needed. In
addition, as noted in recent analysis of simulations (e.g., Weller et al.
2023), environment is also likely to play an important role in determining
where accreting sources fall in relation to the local $M_{\rm BH}-M_{*}$
relation. A recent study by Inayoshi & Ichikawa (2024) suggests that dust-rich
environments in the high-redshift Universe could create conditions prone to
generate intrinsically overmassive populations of BHs, with a distribution
similar to what our data at $z<3$ is showing. Future follow-up observational
studies to characterize the immediate environments for signs of mergers and
interactions would enable us to derive a full picture of the many competing
effects that operate in co-evolution.

## Acknowledgments

The authors thank the anonymous referee for insightful comments. The authors
thank Richard Mushotzky for providing important information regarding the
X-ray detection of some of the sources. The authors would like to express
their appreciation to Jen Miller at Gemini Observatory for providing reduced
GNIRS data for one of our targets. M.M. acknowledges support from the Spanish
Ministry of Science and Innovation through the project PID2021-124243NB-C22.
This work was partially supported by the program Unidad de Excelencia María de
Maeztu CEX2020-001058-M. F.P. acknowledges support from a Clay Fellowship
administered by the Smithsonian Astrophysical Observatory. This work was also
supported by the Black Hole Initiative at Harvard University, which is funded
by grants from the John Templeton Foundation and the Gordon and Betty Moore
Foundation. HS is supported by the international Gemini Observatory, a program
of NSF’s NOIRLab, which is managed by the Association of Universities for
Research in Astronomy (AURA) under a cooperative agreement with the National
Science Foundation, on behalf of the Gemini partnership of Argentina, Brazil,
Canada, Chile, the Republic of Korea, and the United States of America. PN
acknowledges support from the Gordon and Betty Moore Foundation and the John
Templeton Foundation that fund the Black Hole Initiative (BHI) at Harvard
University where she serves as one of the PIs. This work has been supported by
the Polish National Agency for Academic Exchange (Bekker grant
BPN/BEK/2021/1/00298/DEC/1), the European Union’s Horizon 2020 Research and
Innovation programme under the Maria Sklodowska-Curie grant agreement (No.
754510).

Based on observations obtained at the international Gemini Observatory, a
program of NSF’s NOIRLab, GN-2021A-FT-216, which is managed by the Association
of Universities for Research in Astronomy (AURA) under a cooperative agreement
with the National Science Foundation on behalf of the Gemini Observatory
partnership: the National Science Foundation (United States), National
Research Council (Canada), Agencia Nacional de Investigación y Desarrollo
(Chile), Ministerio de Ciencia, Tecnología e Innovación (Argentina),
Ministério da Ciência, Tecnologia, Inovações e Comunicações (Brazil), and
Korea Astronomy and Space Science Institute (Republic of Korea). This work was
enabled by observations made from the Gemini North telescope, located within
the Maunakea Science Reserve and adjacent to the summit of Maunakea. We are
grateful for the privilege of observing the Universe from a place that is
unique in both its astronomical quality and its cultural significance.

## References

  * Agarwal et al. (2013) Agarwal, B., Davis, A. J., Khochfar, S., Natarajan, P., & Dunlop, J. S. 2013, MNRAS, 432, 3438
  * Aihara et al. (2022) Aihara, H., AlSayyad, Y., Ando, M., et al. 2022, PASJ, 74, 247
  * Alexander & Natarajan (2014) Alexander, T., & Natarajan, P. 2014, Science, 345, 1330
  * Baron & Ménard (2019) Baron, D., & Ménard, B. 2019, MNRAS, 487, 3404
  * Beckmann et al. (2023) Beckmann, R. S., Dubois, Y., Volonteri, M., et al. 2023, MNRAS, 523, 5610
  * Bennert et al. (2011) Bennert, V. N., Auger, M. W., Treu, T., Woo, J.-H., & Malkan, M. A. 2011, ApJ, 742, 107
  * Bernardi et al. (2007) Bernardi, M., Sheth, R. K., Tundo, E., & Hyde, J. B. 2007, ApJ, 660, 267
  * Bhowmick et al. (2024) Bhowmick, A. K., Blecha, L., Torrey, P., et al. 2024, arXiv:2402.03626, doi:10.48550/arXiv.2402.03626
  * Birchall et al. (2020) Birchall, K. L., Watson, M. G., & Aird, J. 2020, MNRAS, 492, 2268
  * Birchall et al. (2022) Birchall, K. L., Watson, M. G., Aird, J., & Starling, R. L. C. 2022, MNRAS, 510, 4556
  * Bogdán et al. (2024) Bogdán, Á., Goulding, A. D., Natarajan, P., et al. 2024, Nature Astronomy, 8, 126
  * Boquien et al. (2019) Boquien, M., Burgarella, D., Roehlly, Y., et al. 2019, A&A, 622, A103
  * Burke et al. (2024) Burke, C. J., Liu, Y., Ward, C. A., et al. 2024, arXiv:2402.06882
  * Burke et al. (2023) Burke, C. J., Shen, Y., Liu, X., et al. 2023, MNRAS, 518, 1880
  * Bykov et al. (2024) Bykov, S. D., Gilfanov, M. R., & Sunyaev, R. A. 2024, MNRAS, 527, 1962
  * Calistro Rivera et al. (2016) Calistro Rivera, G., Lusso, E., Hennawi, J. F., & Hogg, D. W. 2016, ApJ, 833, 98
  * Caplar et al. (2018) Caplar, N., Lilly, S. J., & Trakhtenbrot, B. 2018, ApJ, 867, 148
  * Cappelluti et al. (2022) Cappelluti, N., Hasinger, G., & Natarajan, P. 2022, ApJ, 926, 205
  * Chiappetti et al. (2018) Chiappetti, L., Fotopoulou, S., Lidman, C., et al. 2018, A&A, 620, A12
  * Cisternas et al. (2011) Cisternas, M., Jahnke, K., Bongiorno, A., et al. 2011, ApJ, 741, L11
  * Davis et al. (2022) Davis, F., Kaviraj, S., Hardcastle, M. J., et al. 2022, MNRAS, 511, 4109
  * Decarli et al. (2010) Decarli, R., Falomo, R., Treves, A., et al. 2010, MNRAS, 402, 2453
  * Devecchi & Volonteri (2009) Devecchi, B., & Volonteri, M. 2009, ApJ, 694, 302
  * Ding et al. (2020) Ding, X., Silverman, J., Treu, T., et al. 2020, ApJ, 888, 37
  * Duras et al. (2020) Duras, F., Bongiorno, A., Ricci, F., et al. 2020, A&A, 636, A73
  * Eilers et al. (2024) Eilers, A.-C., Mackenzie, R., Pizzati, E., et al. 2024, arXiv:2403.0798s, 2403.07986
  * Fan et al. (2023) Fan, X., Bañados, E., & Simcoe, R. A. 2023, ARA&A, 61, 373
  * Ferrarese & Merritt (2000) Ferrarese, L., & Merritt, D. 2000, ApJ, 539, L9
  * Ferré-Mateu et al. (2018) Ferré-Mateu, A., Forbes, D. A., Romanowsky, A. J., Janz, J., & Dixon, C. 2018, MNRAS, 473, 1819
  * Ferré-Mateu et al. (2021) Ferré-Mateu, A., Mezcua, M., & Barrows, R. S. 2021, MNRAS, 506, 4702
  * Furtak et al. (2024) Furtak, L. J., Labbé, I., Zitrin, A., et al. 2024, Nature, 628, 57
  * Gallo & Sesana (2019) Gallo, E., & Sesana, A. 2019, ApJ, 883, L18
  * Gebhardt et al. (2000) Gebhardt, K., Bender, R., Bower, G., et al. 2000, ApJ, 539, L13
  * Greene & Ho (2005) Greene, J. E., & Ho, L. C. 2005, ApJ, 630, 122
  * Greene et al. (2020) Greene, J. E., Strader, J., & Ho, L. C. 2020, ARA&A, 58, 257
  * Greene et al. (2024) Greene, J. E., Labbe, I., Goulding, A. D., et al. 2024, ApJ, 964, 39
  * Grier et al. (2013) Grier, C. J., Martini, P., Watson, L. C., et al. 2013, ApJ, 773, 90
  * Guo et al. (2018) Guo, H., Shen, Y., & Wang, S. 2018, PyQSOFit: Python code to fit the spectrum of quasars, , , ascl:1809.008
  * Habouzit et al. (2021) Habouzit, M., Li, Y., Somerville, R. S., et al. 2021, MNRAS, 503, 1940
  * Harikane et al. (2023) Harikane, Y., Zhang, Y., Nakajima, K., et al. 2023, ApJ, 959, 39
  * Inayoshi & Ichikawa (2024) Inayoshi, K., & Ichikawa, K. 2024, arXiv:2402.1470, 2402.14706
  * Inayoshi et al. (2022) Inayoshi, K., Nakatani, R., Toyouchi, D., et al. 2022, ApJ, 927, 237
  * Jahnke et al. (2009) Jahnke, K., Bongiorno, A., Brusa, M., et al. 2009, ApJ, 706, L215
  * Juodžbalis et al. (2024) Juodžbalis, I., Maiolino, R., Baker, W. M., et al. 2024, arXiv:2403.03872s, doi:10.48550/arXiv.2403.03872
  * Kocevski et al. (2023) Kocevski, D. D., Onoue, M., Inayoshi, K., et al. 2023, ApJ, 954, L4
  * Kokorev et al. (2023) Kokorev, V., Fujimoto, S., Labbe, I., et al. 2023, ApJ, 957, L7
  * Kormendy & Ho (2013) Kormendy, J., & Ho, L. C. 2013, ARA&A, 51, 511
  * Koudmani et al. (2022) Koudmani, S., Sijacki, D., & Smith, M. C. 2022, MNRAS, 516, 2112
  * Larson et al. (2023) Larson, R. L., Finkelstein, S. L., Kocevski, D. D., et al. 2023, ApJ, 953, L29
  * Liu et al. (2020) Liu, W., Veilleux, S., Canalizo, G., et al. 2020, ApJ, 905, 166
  * Lodato & Natarajan (2006) Lodato, G., & Natarajan, P. 2006, MNRAS, 371, 1813
  * Loeb & Rasio (1994) Loeb, A., & Rasio, F. A. 1994, ApJ, 432, 52
  * Lusso & Risaliti (2016) Lusso, E., & Risaliti, G. 2016, ApJ, 819, 154
  * Madau & Dickinson (2014) Madau, P., & Dickinson, M. 2014, ARA&A, 52, 415
  * Magorrian et al. (1998) Magorrian, J., Tremaine, S., Richstone, D., et al. 1998, AJ, 115, 2285
  * Maiolino et al. (2023) Maiolino, R., Scholtz, J., Curtis-Lake, E., et al. 2023, arXiv:2308.01230, doi:10.48550/arXiv.2308.01230
  * Maiolino et al. (2024) Maiolino, R., Scholtz, J., Witstok, J., et al. 2024, Nature, 627, 59
  * Manzano-King et al. (2019) Manzano-King, C. M., Canalizo, G., & Sales, L. V. 2019, ApJ, 884, 54
  * Merloni et al. (2010) Merloni, A., Bongiorno, A., Bolzonella, M., et al. 2010, ApJ, 708, 137
  * Merritt & Milosavljević (2005) Merritt, D., & Milosavljević, M. 2005, Living Reviews in Relativity, 8, 8
  * Mezcua (2017) Mezcua, M. 2017, International Journal of Modern Physics D, 26, 1730021
  * Mezcua (2019) —. 2019, Nature Astronomy, 3, 6
  * Mezcua et al. (2016) Mezcua, M., Civano, F., Fabbiano, G., Miyaji, T., & Marchesi, S. 2016, ApJ, 817, 20
  * Mezcua et al. (2018a) Mezcua, M., Civano, F., Marchesi, S., et al. 2018a, MNRAS, 478, 2576
  * Mezcua & Domínguez Sánchez (2020) Mezcua, M., & Domínguez Sánchez, H. 2020, ApJ, 898, L30
  * Mezcua & Domínguez Sánchez (2024) —. 2024, MNRAS, arXiv:2401.15152
  * Mezcua et al. (2018b) Mezcua, M., Hlavacek-Larrondo, J., Lucey, J. R., et al. 2018b, MNRAS, 474, 1342
  * Mezcua et al. (2023) Mezcua, M., Siudek, M., Suh, H., et al. 2023, ApJ, 943, L5
  * Mezcua et al. (2019) Mezcua, M., Suh, H., & Civano, F. 2019, MNRAS, 488, 685
  * Mullaney et al. (2012) Mullaney, J. R., Daddi, E., Béthermin, M., et al. 2012, ApJ, 753, L30
  * Natarajan (2004) Natarajan, P. 2004, in Astrophysics and Space Science Library, Vol. 308, Supermassive Black Holes in the Distant Universe, ed. A. J. Barger, 127
  * Natarajan (2014) Natarajan, P. 2014, General Relativity and Gravitation, 46, 1702
  * Natarajan et al. (2024) Natarajan, P., Pacucci, F., Ricarte, A., et al. 2024, ApJ, 960, L1
  * Onken et al. (2004) Onken, C. A., Ferrarese, L., Merritt, D., et al. 2004, ApJ, 615, 645
  * Pacucci & Loeb (2022) Pacucci, F., & Loeb, A. 2022, MNRAS, 509, 1885
  * Pacucci & Loeb (2024) —. 2024, ApJ, 964, 154
  * Pacucci et al. (2021) Pacucci, F., Mezcua, M., & Regan, J. A. 2021, ApJ, 920, 134
  * Pacucci et al. (2023) Pacucci, F., Nguyen, B., Carniani, S., Maiolino, R., & Fan, X. 2023, ApJ, 957, L3
  * Polimera et al. (2022) Polimera, M. S., Kannappan, S. J., Richardson, C. T., et al. 2022, ApJ, 931, 44
  * Portegies Zwart et al. (1999) Portegies Zwart, S. F., Makino, J., McMillan, S. L. W., & Hut, P. 1999, A&A, 348, 117
  * Power et al. (2010) Power, C., Baugh, C. M., & Lacey, C. G. 2010, MNRAS, 406, 43
  * Reines (2022) Reines, A. E. 2022, Nature Astronomy, 6, 26
  * Reines et al. (2020) Reines, A. E., Condon, J. J., Darling, J., & Greene, J. E. 2020, ApJ, 888, 36
  * Reines et al. (2013) Reines, A. E., Greene, J. E., & Geha, M. 2013, ApJ, 775, 116
  * Reines & Volonteri (2015) Reines, A. E., & Volonteri, M. 2015, ApJ, 813, 82
  * Ricarte & Natarajan (2018) Ricarte, A., & Natarajan, P. 2018, MNRAS, 481, 3278
  * Richards et al. (2006) Richards, G. T., Lacy, M., Storrie-Lombardi, L. J., et al. 2006, ApJS, 166, 470
  * Salehirad et al. (2022) Salehirad, S., Reines, A. E., & Molina, M. 2022, ApJ, 937, 7
  * Schutte & Reines (2022) Schutte, Z., & Reines, A. E. 2022, Nature, 601, 329
  * Scodeggio et al. (2018) Scodeggio, M., Guzzo, L., Garilli, B., et al. 2018, A&A, 609, A84
  * Scoggins et al. (2023) Scoggins, M. T., Haiman, Z., & Wise, J. H. 2023, MNRAS, 519, 2155
  * Scoville et al. (2007) Scoville, N., Aussel, H., Brusa, M., et al. 2007, ApJS, 172, 1
  * Setoguchi et al. (2021) Setoguchi, K., Ueda, Y., Toba, Y., & Akiyama, M. 2021, ApJ, 909, 188
  * Shankar et al. (2016) Shankar, F., Bernardi, M., Sheth, R. K., et al. 2016, MNRAS, 460, 3119
  * Shankar et al. (2019) Shankar, F., Bernardi, M., Richardson, K., et al. 2019, MNRAS, 485, 1278
  * Shen & Liu (2012) Shen, Y., & Liu, X. 2012, ApJ, 753, 125
  * Shen et al. (2019) Shen, Y., Hall, P. B., Horne, K., et al. 2019, ApJS, 241, 34
  * Sicilia et al. (2022) Sicilia, A., Lapi, A., Boco, L., et al. 2022, ApJ, 924, 56
  * Silk et al. (2024) Silk, J., Begelman, M. C., Norman, C., Nusser, A., & Wyse, R. F. G. 2024, ApJ, 961, L39
  * Silk & Rees (1998) Silk, J., & Rees, M. J. 1998, A&A, 331, L1
  * Siudek et al. (2023) Siudek, M., Mezcua, M., & Krywult, J. 2023, MNRAS, 518, 724
  * Stone et al. (2023) Stone, M. A., Lyu, J., Rieke, G. H., & Alberts, S. 2023, ApJ, 953, 180
  * Suh et al. (2020) Suh, H., Civano, F., Trakhtenbrot, B., et al. 2020, ApJ, 889, 32
  * Suh et al. (2019) Suh, H., Civano, F., Hasinger, G., et al. 2019, ApJ, 872, 168
  * Sun et al. (2015) Sun, M., Trump, J. R., Brandt, W. N., et al. 2015, ApJ, 802, 14
  * Trakhtenbrot & Netzer (2010) Trakhtenbrot, B., & Netzer, H. 2010, MNRAS, 406, L35
  * Trinca et al. (2022) Trinca, A., Schneider, R., Valiante, R., et al. 2022, MNRAS, 511, 616
  * Übler et al. (2023) Übler, H., Maiolino, R., Curtis-Lake, E., et al. 2023, A&A, 677, A145
  * van Son et al. (2019) van Son, L. A. C., Barber, C., Bahé, Y. M., et al. 2019, MNRAS, 485, 396
  * Vestergaard & Peterson (2006) Vestergaard, M., & Peterson, B. M. 2006, ApJ, 641, 689
  * Volonteri (2010) Volonteri, M. 2010, A&A Rev., 18, 279
  * Volonteri (2012) —. 2012, Sci, 337, 544
  * Volonteri et al. (2008) Volonteri, M., Lodato, G., & Natarajan, P. 2008, MNRAS, 383, 1079
  * Volonteri & Rees (2005) Volonteri, M., & Rees, M. J. 2005, ApJ, 633, 624
  * Webb et al. (2020) Webb, N. A., Coriat, M., Traulsen, I., et al. 2020, A&A, 641, A136
  * Weller et al. (2023) Weller, E. J., Pacucci, F., Natarajan, P., & Di Matteo, T. 2023, MNRAS, 522, 4963
  * Whitaker et al. (2014) Whitaker, K. E., Franx, M., Leja, J., et al. 2014, ApJ, 795, 104
  * Yang et al. (2020) Yang, G., Boquien, M., Buat, V., et al. 2020, MNRAS, 491, 740
  * Yue et al. (2023) Yue, M., Eilers, A.-C., Simcoe, R. A., et al. 2023, arXiv:2309.04614, doi:10.48550/arXiv.2309.04614
  * Zhuang & Ho (2023) Zhuang, M.-Y., & Ho, L. C. 2023, Nature Astronomy, 7, 1376
  * Ziegler & Freese (2022) Ziegler, J., & Freese, K. 2022, arXiv:2212.13903, doi:10.48550/arXiv.2212.13903

Figure 4: One of the 12 ‘cosmic noon’ low-mass galaxies hosting an overmassive
BH. Top left: Spectral energy distribution fitting of the rest-frame observed
ultraviolet, optical, and infrared (when available) photometry (black points)
with the best-fit model (black curve), including a combination of the galaxy
template (green), an AGN accretion disk component (blue) and an AGN dust torus
model (yellow). Top right: PDF for the stellar mass taking into account all
possible fractions of AGN emission and providing an upper limit on the stellar
mass. So the most probable value (MsPDF, blue dashed line) has a higher value
than the best-fit stellar mass (MsBEST, red solid line). The 16 and 84
percentile intervals (gray shades) are also indicated. Middle: Emission line
fitting of the VIPERS spectrum including the continuum emission (in yellow,
top panel), and the broad lines (in blue) decomposed into broad (in red) and
narrow (in green) components (zoom-in in the bottom panels). Bottom: Subaru
Hyper Suprime-Cam image in the i-band.

##  Appendix A Gemini observations

We obtained a 0.8-2.5$\mu$m spectrum of 401126746 using the Gemini Near-
Infrared Spectrograph (GNIRS) on the Gemini telescope with queue-observing
mode, under the program GN-2021A-FT-216. GNIRS was configured using a Short
Blue camera (0,15”/pixel), 32 l/mm cross-dispersed mode, and a 0.675” wide
slit. Observations utilized the standard ABBA method of nodding along the slit
to enable sky subtraction. We also observed telluric standard stars before and
after the observations. The data were reduced using the XDGNIRS pipeline
developed by the Gemini Observatory, based on the Gemini IRAF package. The
reduction pipeline includes standard image cleaning for pattern noise and
artifacts, flatfielding, sky subtraction, distortion correction, and
rectification of 2D data. The arc spectra were used for wavelength
calibration. The spectrum of the telluric standards was processed in a similar
way, followed by the removal of intrinsic hydrogen absorption lines, and used
for the telluric corrections and flux calibration.


# Reducing The Mismatch Between Marginal and Learned Distributions in Neural
Video Compression

Muhammet Balcilar  InterDigital, Inc.
Rennes, France
Bharath Bhushan Damodaran  InterDigital, Inc.
Rennes, France
Pierre Hellier  InterDigital, Inc.
Rennes, France

###### Abstract

During the last four years, we have witnessed the success of end-to-end
trainable models for image compression. Compared to decades of incremental
work, these machine learning (ML) techniques learn all the components of the
compression technique, which explains their actual superiority. However, end-
to-end ML models have not yet reached the performance of traditional video
codecs such as VVC. Possible explanations can be put forward: lack of data to
account for the temporal redundancy, or inefficiency of latent’s density
estimation in the neural model. The latter problem can be defined by the
discrepancy between the latent’s marginal distribution and the learned prior
distribution. This mismatch, known as amortization gap of entropy model,
enlarges the file size of compressed data. In this paper, we propose to
evaluate the amortization gap for three state-of-the-art ML video compression
methods. Second, we propose an efficient and generic method to solve the
amortization gap and show that it leads to an improvement between $2\%$ to
$5\%$ without impacting reconstruction quality.

###### Index Terms:

Neural video compression, Entropy model, Reparameterization.

††publicationid: pubid: 978-1-6654-7592-1/22/$31.00 ©2022 IEEE

##  I Introduction

Image and video compression is a fundamental task in image processing, which
has become crucial in the time of pandemic and increasing video streaming.
Thanks to the community’s huge efforts for decades, traditional methods
(including linear transformations under heavily optimized handcrafted
techniques) have reached current state of the art rate-distortion (RD)
performance and dominate current industrial codecs solutions. Alternatively,
end-to-end trainable deep models have recently emerged, with promising
results. Even though these methods clearly exceed many traditional techniques
and surpass human capability for some tasks for a few years back very recently
they beat the best traditional compressing method (VVC, versatile video coding
[1]) even in terms of peak signal-to-noise ratio (PSNR) for single image
compression [2]. However, their performance on video compression are still far
from VVC, just on par with one generation back of the traditional method
(HEVC, high efficiency video coding [3]). In addition to the inefficiency of
neural models on capturing temporal redundancy, the mismatch between test
latent’s normalized histograms and learned distributions in the entropy models
may be a contributing factor.

Lossy image compression via end-to-end trainable models is special kind of
Variational Autoencoder (VAE) that learns the transformations among data and
latent codes and the probability models of these latent codes jointly [4, 5,
6, 7, 2]. The problem is a multi-objective optimization problem where the
model should be optimized for reconstruction quality and cross entropy of
latent code w.r.t learned probabilities known as RD loss function. These
neural image codecs were extended by using two VAEs, one for encoding motion
information, another for encoding residual information in end-to-end video
compression [8, 9, 10, 11, 12]. As all trainable models suffer from
amortization gap [13] (may be optimal for entire dataset, but sub-optimal for
given single test instance), neural compression models have a similar issue
and this gap reduces the performance by either enlarging the file size or
decreasing the reconstruction quality. The first fold solution to this problem
is to apply post training for a given single test image/video, where some of
them train just the encoder component of VAE [14, 15] in order to prevent
extra signaling cost. The other selection is to fine tune all parts of the
model by adding signaling cost to the loss function [16]. Second fold
solutions do not apply time-consuming post training, but adjust some
parameters of the model. For instance, in [17], only the entropy model’s
amortization gap in end-to-end image compression is targeted and instance
specific reparameterization of the latent distribution was proposed.

This paper is an extension of our previous work [17] to the video compression
with some important differences. First, we introduce a general framework that
generalizes end-to-end trainable models on video compression. Second, we
analyse the amortization gap of all entropy models for different frames (I, B
and P frames), for different information (motion and residual) in three recent
neural video compression methods. Third, we identify the origin of the main
performance drop and we show how it can be fixed. Last, but not least, we show
the efficiency of our probability reparameterization method where the new
parameters are kept into file considering the temporal redundancy of these
parameters. According to the result, we decrease the file size of video
between $2\%$ to $5\%$ in average without any effect on reconstruction
quality. To the best of our knowledge, it is the first research on closing the
amortization gap of neural video compression without post-training.

##  II End-to-End Video Compression

Although the first end-to-end image compression model used only factorized
entropy model in [4], following papers with hierarchical VAE based hyperprior
entropy models as in [5, 6, 7, 2] became standard in neural image compression
and now form the backbone of sota neural video compression. Let
$\mathbf{x},\mathbf{\bar{x}},\mathbf{\hat{x}},\mathbf{\hat{x}}_{r}$
respectively denote the current image, motion warped current image, the
reconstruction of current image and the reconstruction of reference image. Let
us note $\mathbf{v}$ as the predicted motion information,
$\mathbf{y}_{m},\mathbf{\hat{y}}_{m},\mathbf{y}_{r},\mathbf{\hat{y}}_{r}$
continuous latent, quantized (or noise added) latent of motion information and
residual information respectively. Similarly, we will note
$\mathbf{z}_{m},\mathbf{\hat{z}}_{m},\mathbf{z}_{r},\mathbf{\hat{z}}_{r}$ as
the continuous side latent and quantized (or noise added) side latent of
motion information and residual information respectively. $\mathbf{Q}(.)$
element-wise function applies quantization in test time or its continuous
relaxation in train time as $\mathbf{Q}(x)=x+\epsilon$ that $\epsilon\sim
U(-0.5,0.5)$ and $\mathbf{W}(.,.)$ warps given image w.r.t given motion. First
VAE whose aim is to encode motion information takes current frame and
reconstructed reference frame (or in B frame encoding, 2 reference frames) as
inputs and finds warped image by
$\mathbf{y}_{m}=gm_{a}(\mathbf{x},\mathbf{\hat{x}}_{r};\mathbf{\phi}_{m})$,
$\mathbf{\hat{y}}_{m}=\mathbf{Q}(\mathbf{y}_{m})$,
$\mathbf{v}=gm_{s}(\mathbf{\hat{y}_{m}};\mathbf{\theta}_{m})$,
$\mathbf{z}_{m}=hm_{a}(\mathbf{y}_{m};\Phi_{m})$,
$\mathbf{\hat{z}}_{m}=\mathbf{Q}(\mathbf{z}_{m})$, and
$\mathbf{\bar{x}}=\mathbf{W}(\mathbf{\hat{x}}_{r},\mathbf{v})$. Second VAE is
for encoding residual information (difference between current frame and its
motion warped version) and reconstruction of the image by applying
$\mathbf{y}_{r}=gr_{a}(\mathbf{x}-\mathbf{\bar{x}};\mathbf{\phi}_{r})$,
$\mathbf{\hat{y}}_{r}=\mathbf{Q}(\mathbf{y}_{r})$,
$\mathbf{z}_{r}=hr_{a}(\mathbf{y}_{r};\Phi_{r})$,
$\mathbf{\hat{z}}_{r}=\mathbf{Q}(\mathbf{z}_{r})$ and
$\mathbf{\hat{x}}=gr_{s}(\mathbf{\hat{y}_{r}};\mathbf{\theta}_{r})+\mathbf{\bar{x}}$.
As long as $gm_{a},gm_{s},hm_{a},gr_{a},gr_{s},hr_{a}$ are trainable deep
models, neural video compression loss can be written in followings;

$\scriptstyle\mathcal{L}=\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{x},\mathbf{\hat{x}}_{r}\sim
p_{x}\\\ \epsilon\sim
U\end{subarray}}[-log(p^{(m)}_{h}(\mathbf{\hat{y}}_{m}|\mathbf{\hat{z}}_{m},\mathbf{\Theta}_{m}))-log(p^{(m)}_{f}(\mathbf{\hat{z}}_{m}|\Psi_{m}))\\\
\scriptstyle-
log(p^{(r)}_{h}(\mathbf{\hat{y}}_{r}|\mathbf{\hat{z}}_{r},\mathbf{\Theta}_{r}))-log(p^{(r)}_{f}(\mathbf{\hat{z}}_{r}|\Psi_{r}))+\lambda.d(\mathbf{x},\mathbf{\hat{x}})],$
(1)

where $p^{(m)}_{f}(.|\Psi_{m})$ and $p^{(r)}_{f}(.|\Psi_{r})$ are factorized
entropy model for motion and residual information,
$p^{(m)}_{h}(.|\mathbf{\hat{z},\Theta_{m}})$ and
$p^{(r)}_{h}(.|\mathbf{\hat{z},\Theta_{r}})$ are hyperprior entropy models for
motion and residual information respectively implemented with neural networks,
$d(.,.)$ is any distortion loss such as MSE for PSNR metric, $\lambda$
hyperparameter that plays trade-off role between compression ratio and
quality. Here the variables to be written into compressed file are motion’s
main and side information ($\mathbf{\hat{y}}_{m},\mathbf{\hat{z}}_{m}$) and
residual’s main and side information
($\mathbf{\hat{y}}_{r},\mathbf{\hat{z}}_{r}$) whose expected file sizes under
learned entropy models are the first four parts of the loss in (1).

Factorized entropy models learn the probability mass function (pmf) of the
symbols for each feature band of $\mathbf{\hat{z}}_{m}$ and
$\mathbf{\hat{z}}_{r}$ separately. Thus, learned pmf values are defined by
parameters of factorized entropy model $\Psi_{m},\Psi_{r}$. If the side latent
of motion or residual information is
$\mathbf{\hat{z}}_{m|r}\in\mathbf{R}^{k\times k\times f}$ and
$\hat{p}_{\Psi_{m|r}}^{(c)}(.)$ is the pmf table of $c$-th feature band for
motion or residual information, the factorized entropy model applies;

$p^{(m|r)}_{f}(\mathbf{\hat{z}}_{m|r}|\Psi_{m|r})=\prod_{c=1}^{f}\prod_{i,j=1}^{k,k}\hat{p}_{\Psi_{m|r}}^{(c)}({\mathbf{\hat{z}}_{m|r,i,j,c}}).$
(2)

Hyperprior entropy model learns the parameters of main information’s
probability function (usually gaussian or laplacian distribution is used)
using already encoded side latent by
$\mu,\sigma=hm_{s}(\mathbf{\hat{z}}_{m};\mathbf{\Theta}_{m})$ or
$\mu,\sigma=hr_{s}(\mathbf{\hat{z}}_{r};\mathbf{\Theta}_{r})$ for motion or
residual information and applies as follows;

$p^{(m|r)}_{h}(\mathbf{\hat{y}}_{m|r}|\mathbf{\hat{z}}_{m|r},\mathbf{\Theta}_{m|r})=\prod_{c=1}^{s}\prod_{i\in\mathcal{N}(\sigma_{c})}\hat{N}(\mathbf{\tilde{y}}_{m|r,i};0,\sigma_{c}),$
(3)

where $\mathbf{\tilde{y}}_{m|r,i}=Q(\mathbf{y}_{m|r,i}-\mu_{i})$,
$\mathbf{\hat{y}}_{m|r,i}=\mathbf{\tilde{y}}_{m|r,i}+\mu_{i}$, $\sigma_{c}$ is
$c$-th predefined scale, $\mathcal{N}(\sigma_{c})$ is a set of latent index
whose winning scale is $\sigma_{c}$, $\hat{N}(x;\mu,\sigma)$ pmf value of $x$
for 1-d gaussian distribution with $\mu,\sigma$ parameters, $s$ is number of
predefined scale values of gaussian distribution and $hm_{s},hr_{s}$ are
another deep neural networks.

In this framework, end-to-end video compression consists of $5$ trainable
components for motion information $gm_{a},gm_{s},hm_{a},hm_{s},p^{(m)}_{f}$
and 5 components for residual information
$gr_{a},gr_{s},hr_{a},hr_{s},p^{(r)}_{f}$ parameterized by
$\mathbf{\phi}_{m},\mathbf{\theta}_{m},\Phi_{m},\mathbf{\Theta}_{m},\Psi_{m}$
and
$\mathbf{\phi}_{r},\mathbf{\theta}_{r},\Phi_{r},\mathbf{\Theta}_{r},\Psi_{r}$
respectively and a non-trainable motion warping function $\mathbf{W}(.,.)$.
The selection of all these $11$ components explains the differences between
studied end-to-end video compression methods.

##  III Method

The gap caused by the mismatch between marginal and learned distribution and
our proposals are given in this section.

###  III-A Gap of the Entropy Models

Figure 1: Learned pmfs in the model
($\hat{p}_{{\Psi}_{m}},\hat{N},\hat{p}_{{\Psi}_{r}},\hat{N}$), pmf after our
reparameterization
($\tilde{p}^{(m)}_{f},\tilde{p}^{(m)}_{h},\tilde{p}^{(r)}_{f},\tilde{p}^{(r)}_{h}$)
and normalized frequencies ($h^{(m)}_{f},h^{(m)}_{h},h^{(r)}_{f},h^{(r)}_{h}$)
for a certain image’s selected latents of motion’s side, motion’s main,
residual’s side and residual’s main information respectively. Our
reparameterization fits better on the normalized frequencies, thus it
compresses better.

When considering intra frame encoding (I frame, as single image encoding)
there is no motion nor reference images, thus first two component of the loss
in (1) is canceled out and $\mathbf{\bar{x}}=0$. Consequently, only the
residual’s main and side information have to be encoded for I frames. However,
for inter frame encoding (either B or P), both motion’s and residual’s main
and side information are encoded. Using entropy models in (2) and (3), we can
write expected bitlength of $c$-th feature band of side information as
follows:

$l^{(m|r)}_{f,c}=-\sum_{i=1}^{k}\sum_{j=1}^{k}log\left(\hat{p}_{\Psi_{m|r}}^{(c)}({\mathbf{\hat{z}}_{m|r,i,j,c}})\right).$
(4)

We can write expected bitlength of main information that represented by $c$-th
predefined scale as follows;

$l^{(m|r)}_{h,c}=-\sum_{i\in\mathcal{N}(\sigma_{c})}log\left(\hat{N}(\mathbf{\tilde{y}}_{m|r,i};0,\sigma_{c})\right).$
(5)

Considering all side information’s bitlength, it can be calculated by
$l^{(m|r)}_{f}=\sum_{c}(l^{(m|r)}_{f,c})$ and
$l^{(m|r)}_{h}=\sum_{c}(l^{(m|r)}_{h,c})$ for all main information’s
bitlength. This leads us to calculate baseline method’s bitlength of inter
frame by summing up all 4 informations’ bitlength by
$l^{(b)}=l^{(m)}_{f}+l^{(m)}_{h}+l^{(r)}_{f}+l^{(r)}_{h}$.

The optimality of bit length of each information depends on how learned pmfs
($\hat{p}_{\Psi_{m}},\hat{p}_{\Psi_{r}},\hat{N}(\mathbf{\tilde{y}}_{m};0,\sigma_{c}),\hat{N}(\mathbf{\tilde{y}}_{r};0,\sigma_{c})$
are closed to marginal distribution of latents
($\mathbf{\hat{z}}_{m},\mathbf{\hat{z}}_{r},\mathbf{\hat{y}}_{m},\mathbf{\hat{y}}_{r}$)
and it is defined as amortization gap of entropy models in neural image
compression [17]. This mismatch can be seen by differences between green
curves and blue histogram bars in Fig 1. Following the same procedure, we can
calculate the theoretical limit of expected bit length of each information by
simply replacing learned pmf by corresponding latent’s normalized histogram as
follows;

$l^{*(m|r)}_{f,c}=-\sum_{i,j=1}^{k,k}log\left(h_{f,c}^{(m|r)}({\mathbf{\hat{z}}_{m|r,i,j,c}})\right),$
(6)

where
$h_{f,c}^{(m|r)}(x)=\sum_{i,j}\delta(x,\mathbf{\hat{z}}_{m|r,i,j,c})/k^{2}$ is
normalized frequency of symbol $x$ on $k\times k$ slice of
$\mathbf{\hat{z}}_{m|r}$ and $\delta(.,.)$ is Kronecker delta. Theoretical
limit of expected bit length of main information can be written as follows;

$l^{*(m|r)}_{h,c}=-\sum_{i\in\mathcal{N}(\sigma_{c})}log\left(h_{h,c}^{(m|r)}(\mathbf{\tilde{y}}_{m|r,i})\right),$
(7)

where
$h_{h,c}^{(m|r)}(x)=\sum_{i\in\mathcal{N}(\sigma_{c})}\delta(x,\mathbf{\tilde{y}}_{m|r,i})/|\mathcal{N}(\sigma_{c})|$
is normalized frequency of symbol $x$ on $\mathbf{\tilde{y}}_{m|r,i}$ where
$i\in\mathcal{N}(\sigma_{c})$ and $|\mathcal{N}(\sigma_{c})|$ is the number of
element in the given set. Following the same procedure, bitlength of all side
and main information can be calculated by
$l^{*(m|r)}_{f}=\sum_{c}(l^{*(m|r)}_{f,c})$ and
$l^{*(m|r)}_{h}=\sum_{c}(l^{*(m|r)}_{h,c})$. Thus, theoretical limit of
bitlength of inter frame would be
$l^{*}=l^{*(m)}_{f}+l^{*(m)}_{h}+l^{*(r)}_{f}+l^{*(r)}_{h}$. The differences
between baseline model’s each information’s bitlength and theoretical limits
of the bitlength gives the amortization gap of corresponding type of
information.

Algorithm 1 Encoding with Factorized Entropy

Input: Learned pmfs $\hat{p}_{\Psi}^{(c)}$ for $c=1\dots s$, latents
$\mathbf{\hat{z}}\in\mathbb{R}^{k\times k\times s}$, number of mixture $K$ and
previous inter frame’s params $\hat{\beta}_{p}^{*(c)}$ for $c=1\dots s$.

Output: latent bitstream $lb$ and parameter bitstream $pb$.

Reorder $\hat{p}_{\Psi}^{(c)}$ for $c=1\dots s$ w.r.t its entropy

for $c=1$ to $S$ do

$\beta^{*(c)}=argmax_{\beta}\sum_{i,j}log\left(\tilde{p}(\mathbf{\hat{z}}_{i,j,c});\beta^{(c)})\right)$
// find parameters

$\hat{\beta}^{*(c)}=Q(\beta^{*(c)})$ // quantize the best parameters

$\mathcal{G}=\sum_{i,j}-log_{2}(\hat{p}_{\Psi}^{(c)}(\mathbf{\hat{z}}_{i,j,c}))+log_{2}(\tilde{p}(\mathbf{\hat{z}}_{i,j,c};\hat{\beta}^{*(c)}))$
// gain

if $\hat{\beta}_{p}^{*(c)}=None$ and $\mathcal{G}<=30K-10$ then

$pb.write(True)$ // write temporal bit

$\hat{\beta}^{*(c)}=None$ // set best params None

$lb.write(\mathbf{\hat{z}}_{:,c},\hat{p}_{\Psi}^{(c)})$ // write latent w.r.t
learned pmf

else if $\hat{\beta}^{*(c)}=\hat{\beta}_{p}^{*(c)}$ and $\mathcal{G}>0$ then

$pb.write(True)$ // write temporal bit

$lb.write(\mathbf{\hat{z}}_{:,c},\tilde{p}(.;\hat{\beta}_{p}^{*(c)}))$ //
write latent w.r.t previous pmf

else if $\mathcal{G}>30K-10$ then

$pb.write(False)$ // write temporal bit

$pb.write(True)$ // write replacement bit

$pb.write(\hat{\beta}^{*(c)})$ // write parameters explicitly

$lb.write(\mathbf{\hat{z}}_{:,c},\tilde{p}(.;\hat{\beta}^{*(c)}))$ // write
latent w.r.t new pmf

else

$pb.write(False)$ // write temporal bit

$pb.write(False)$ // write replacement bit

$\hat{\beta}^{*(c)}=None$ // set best params None

$lb.write(\mathbf{\hat{z}}_{:,c},\hat{p}_{\Psi}^{(c)})$ // write latent w.r.t
learned pmf

end if

end for

for $c=S+1$ to $s$ do

$lb.write(\mathbf{\hat{z}}_{:,c},\hat{p}_{\Psi}^{(c)})$ // write latent w.r.t
learned pmf

end for

###  III-B Temporal Re-parameterization

In order to reduce the mismatch, learned pmfs in the model were replaced by
some parametric distribution $\tilde{p}_{f|h}^{(m|r)}(.;\beta)$ whose
parameters $\beta$, to be signalled to the receiver, are obtained by fitting
the actual histogram of the latents as in [17]. We use the same
reparameterizations that are truncated Gaussian mixture for factorized entropy
(motion/residual’s side information) and truncated zero-mean Gaussian
distribution for hyperprior entropy (motion/residual’s main information) where
the parameters are discretized into 10 bits. Sample probabilities with $K=1$
Gaussian mixture and zero-mean Gaussian can be found in Fig. 1 in the red
curves for all four types of information. When $l^{o(m|r)}_{f|h}$ is the
bitlength of motion/residual’s side/main information under
$\tilde{p}_{f|h}^{(m|r)}(.;\beta)$ probabilities, plus the signaling cost of
$\beta$, the bitlength of interframe in our proposal would be
$l^{(o)}=l^{o(m)}_{f}+l^{o(m)}_{h}+l^{o(r)}_{f}+l^{o(r)}_{h}$.

TABLE I: Ratio of amount of certain information in the bitstream, its amortization gap and our savings for different frame types and 16 length sequence of video. All numbers are percentage and obtained by average results of 7 videos in UVG test video set. All methods are tested for the provided lowest bit rate. Method | Frame | Motion Information | Residual Information | All
---|---|---|---|---
Factorized Entropy (Side) | Hyperprior Entropy (Main) | Factorized Entropy (Side) | Hyperprior Entropy (Main) |
Ratio | Gap | Saving | Ratio | Gap | Saving | Ratio | Gap | Saving | Ratio | Gap | Saving | Gap | Saving
SSF [9] | I | - | - | - | - | - | - | 16.1 | 10.6 | 5.5 | 83.9 | 1.7 | 0.9 | 3.1 | 1.7
P | 5.1 | 22.1 | 11.7 | 2.6 | 6.3 | 0.9 | 15.4 | 11.1 | 5.4 | 76.9 | 1.5 | 0.9 | 4.2 | 2.2
Video | 4.4 | 22.1 | 11.7 | 2.3 | 6.3 | 0.9 | 15.5 | 11.0 | 5.4 | 77.8 | 1.6 | 0.9 | 4.1 | 2.0
LHBDC [11] | I | - | - | - | - | - | - | 10.2 | 14.7 | 11.1 | 89.8 | 1.3 | 0.8 | 2.7 | 1.9
B | 0.4 | 18.0 | 1.6 | 3.5 | 12.0 | 1.9 | 13.2 | 43.9 | 37.2 | 82.9 | 2.6 | 0.8 | 8.3 | 5.6
Video | 0.2 | 18.0 | 1.6 | 2.1 | 12.0 | 1.9 | 12.2 | 33.6 | 27.9 | 85.5 | 2.0 | 0.8 | 6.1 | 4.1
AIVC [10] | I | - | - | - | - | - | - | 6.2 | 38.5 | 34.1 | 93.8 | 1.6 | 0.6 | 3.8 | 2.7
P | 3.5 | 32.7 | 20.0 | 0.4 | 17.5 | 0.0 | 8.2 | 25.1 | 18.3 | 87.9 | 2.3 | 1.0 | 5.3 | 3.0
B | 3.9 | 38.5 | 19.3 | 0.1 | 26.0 | 0.0 | 14.9 | 40.9 | 27.7 | 81.1 | 4.3 | 2.2 | 11.1 | 6.6
Video | 2.4 | 37.5 | 19.4 | 0.1 | 22.7 | 0.0 | 10.8 | 39.0 | 28.2 | 86.7 | 2.9 | 1.4 | 7.7 | 4.7
Figure 2: Amortization gaps and our savings relative to total file size of 3
recent neural video compression methods for 16 frame length video sequence
compression on 7 videos in UVG dataset under different reconstruction quality.

To take temporal redundancies of parameter signaling cost into account, we use
$S$-bit temporal mask to explain if the previously encoded interframe’s
corresponding parameter is the same or not for the top $S$ number of pmf
tables. If this bit is 1, there is no need to encode new parameters but uses
previously encoded interframe’s parameters. In this way, we decrease the
temporal redundancy of these parameters. When considering the signaling cost
of each pmf table, we have seen that just a few of the pmf table is worth to
replace with reparameterization one but vast majority of them is not. To
determine which is worth or not, we reorder the pmf table w.r.t its own
entropy (in hyperprior model, it is not necessary since the pmf tables are
ordered by predefined scale values and lower scales pmf table highly likely
carries more information). Later we test top $S$ number of pmf table one by
one if the expected bit gain is larger than reparameterization cost or not.
The new parameters are written to the file and pmf is replaced if the gain is
larger than that signaling cost. When $K$ number of mixture is used, the
signaling cost is $10(3K-1)$ bits in factorized entropy. Since there is just
one parameter in zero-mean gaussian, the signaling cost is $10$ bits in
hyperprior model when temporal mask is 0. If the parameters are the same with
previously encoded interframe, signaling cost is 0 bit for all entropy models.
To tell the receiver which pmf is replaced, we need to use 1-bit replacement
mask in addition to replace pmf’s parameters and temporal mask. As long as
$\tilde{p}(.;\hat{\beta}^{(c)})$ is the new $c$-th pmf table parameterized by
$\hat{\beta}^{(c)}$ and previous encoded interframe’s parameter is
$\hat{\beta}_{p}^{*(c)}$ (all params of the first interframe, and the pmf
table which are not replaced are $None$), detailed flow of factorized entropy
encoding is given in Algorithm 1. Extending it on hyperprior entropy model is
straightforward by changing only the variable names/indexing.

##  IV Results

In order to test our proposal, we use 7 video sequences in UVG dataset [18] at
1080p resolution. We use 10 set of consecutive 16 frames (total 160 frames) of
each sequence and compress them by compressai [19] implementation of SSF [9],
and author’s official implementations of LHBDC [11] and AIVC [10]. Since all
methods were proposed on different purpose, we do not make comparison of these
methods but measure saving of our generic solution up to these baselines
model. SSF encodes first frame as I frame and rest of the 15 frame as P
frames. LHBDC needs 2 reference frames. We supposed first frame and 17th frame
is I frame and all 15 frames in between frames are B frames encoded by
hierarchical bi-directional settings. In calculations, we did not count 17th
frame, because it is also next GOP’s first reference frame and should be
accounted in next GOP’s file size. AIVC encodes first frame as I frame, 16th
frames as P frames and rest of the in between frames as B frames. In order to
show detailed gap sources, we give the result in Table I for the lowest bpp
objective. In all cases, the ratio is percentage of each information’s
bitlength ($l_{f}^{(m)},l_{h}^{(m)},l_{f}^{(r)},l_{h}^{(r)}$) w.r.t the total
bitlength $l^{(b)}$ in the baseline model. The gap of each information is
$1-l_{f|h}^{*(m|r)}/l_{f|h}^{(m|r)}$ while the total gap is $1-l^{*}/l^{(b)}$.
It can be seen that for all methods, the gap is not negligible since a
potential performance gain of several percent could be achieved. We calculate
our savings by $1-l_{f|h}^{o(m|r)}/l_{f|h}^{(m|r)}$ for each information type,
where the total saving is $1-l^{(o)}/l^{(b)}$. We measure those percentages
for each frame type as well as 16-length sequences of videos reported on the
Table I.

In all models and frame types, the residual’s main information represents the
vast majority of the bitstream ($76.9\%-93.8\%$) whose gap is the smallest
($1.3\%-4.3\%$). Our proposal can fill almost half of this gap. Residual’s
side information is the second largest information type in the interframe’s
bitstream where the biggest gap exists ($10.6\%-43.9\%$). Truncated gaussian
mixture fills more than half of these gaps. Even though the gap of motion
information is larger, the amount of the data is small and its effect on
general performance is limited. Since our reparameterization approach has
fixed cost, if the amount of data is very low (less than $1\%$ in AIVC and
around $2\%$ in SSF’s main motion information and less than $1\%$ in LHBDC’s
side motion information), proposed method cannot save any bits or saving is
very limited even though the gap is $10\%-20\%$. Our performance on side
information (factorized entropy) is much more larger than performance on main
information (hyperprior entropy) for all methods. In average of 16 frame
length video for the lowest bpp objective, the gap is between $4.1\%-7.7\%$
and our saving is between $2.0\%-4.7\%$. To measure the gap and our savings in
different quality/rate, we perform several tests with the pretrained models
provided by authors. Results are given in Fig 2 and show that gap and our
savings decreases with the increment of quality/rate. Even at the highest
quality/rate objective, the gain is better than $1\%$ for all methods.

##  V Conclusion

In this work, we propose a low computational demanding solution for the
inefficiency of entropy coding in neural video compression. Since the method
is generic, it can be applied up to any neural video compression and decrease
the bitlength between $2\%-5\%$ according to our tests. The amortization gap
is inevitable in all ML based models. However, the connection of the amount of
this gap to the certain type of neural codec should be explored in order to
develop better codec which is inside our agenda as a future work.

## References

  * [1] B. Bross, J. Chen, J.-R. Ohm, G. J. Sullivan, and Y.-K. Wang, “Developments in international video coding standardization after avc, with an overview of versatile video coding (vvc),” _Proceedings of the IEEE_ , vol. 109, no. 9, pp. 1463–1493, 2021.
  * [2] G. Gao, P. You, R. Pan, S. Han, Y. Zhang, Y. Dai, and H. Lee, “Neural image compression via attentional multi-scale back projection and frequency decomposition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 677–14 686.
  * [3] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (hevc) standard,” _IEEE Transactions on circuits and systems for video technology_ , vol. 22, no. 12, pp. 1649–1668, 2012\.
  * [4] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in _ICLR_ , 2017.
  * [5] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” in _ICLR_ , 2018.
  * [6] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31, 2018.
  * [7] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in _CVPR_ , 2020.
  * [8] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “Dvc: An end-to-end deep video compression framework,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 006–11 015.
  * [9] E. Agustsson, D. Minnen, N. Johnston, J. Balle, S. J. Hwang, and G. Toderici, “Scale-space flow for end-to-end optimized video compression,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8503–8512.
  * [10] T. Ladune and P. Philippe, “Aivc: Artificial intelligence based video codec,” _arXiv preprint arXiv:2202.04365_ , 2022.
  * [11] M. A. Yılmaz and A. M. Tekalp, “End-to-end rate-distortion optimized learned hierarchical bi-directional video compression,” _IEEE Transactions on Image Processing_ , 2021.
  * [12] R. Pourreza and T. Cohen, “Extending neural p-frame codecs for b-frame coding,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6680–6689.
  * [13] C. Cremer, X. Li, and D. Duvenaud, “Inference suboptimality in variational autoencoders,” _ICML_ , 2018.
  * [14] G. Lu, C. Cai, X. Zhang, L. Chen, W. Ouyang, D. Xu, and Z. Gao, “Content adaptive and error propagation aware deep video compression,” in _ECCV_ , vol. 12347, 2020, pp. 456–472.
  * [15] J. Campos, S. Meierhans, A. Djelouah, and C. Schroers, “Content adaptive optimization for neural image compression,” in _CVPR Workshops_ , June 2019\.
  * [16] T. van Rozendaal, I. A. Huijben, and T. S. Cohen, “Overfitting for fun and profit: Instance-adaptive data compression,” in _ICLR_ , 2021.
  * [17] M. Balcilar, B. Damodaran, and P. Hellier, “Reducing the amortization gap of entropy bottleneck in end-to-end image compression,” in _Picture Coding Symposium (PCS)_ , 2022.
  * [18] A. Mercat, M. Viitanen, and J. Vanne, “Uvg dataset: 50/120fps 4k sequences for video codec analysis and development,” in _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020, pp. 297–302.
  * [19] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” _arXiv preprint arXiv:2011.03029_ , 2020.