# The physical properties of massive green valley galaxies as a function of
environments
at $0.5<z<2.5$ in 3D-HST/CANDELS fields

Wenjun Chang  Institute of Astronomy and Astrophysics, Anqing Normal
University, Anqing 246133, China<EMAIL_ADDRESS>Department of Physics
and Astronomy, University of California, Riverside, 900 University Avenue,
Riverside, CA 92521, USA  Department of Astronomy, University of Science and
Technology of China, Hefei 230026, China<EMAIL_ADDRESS>School of
Astronomy and Space Sciences, University of Science and Technology of China,
Hefei 230026, China  Guanwen Fang  Institute of Astronomy and Astrophysics,
Anqing Normal University, Anqing 246133, China<EMAIL_ADDRESS>Yizhou
Gu  Department of Astronomy, School of Physics and Astronomy, Shanghai Jiao
Tong University, Shanghai 200240, China  Zesen Lin  Department of Astronomy,
University of Science and Technology of China, Hefei 230026, China;
<EMAIL_ADDRESS>School of Astronomy and Space Sciences, University of
Science and Technology of China, Hefei 230026, China  Shiying Lu  School of
Astronomy and Space Science, Nanjing University, Nanjing 210093, People’s
Republic of China  Key Laboratory of Modern Astronomy and Astrophysics
(Nanjing University), Ministry of Education, Nanjing 210093, China  Xu Kong
Department of Astronomy, University of Science and Technology of China, Hefei
230026, China<EMAIL_ADDRESS>School of Astronomy and Space Sciences,
University of Science and Technology of China, Hefei 230026, China

###### Abstract

To investigate the effects of environment in the quenching phase, we study the
empirical relations for green valley (GV) galaxies between overdensity and
other physical properties (i.e., effective radius $r_{\rm e}$, Sérsic indices
$n$, and specific star formation rate sSFR). Based on five 3D-HST/CANDELS
fields, we construct a large sample of 2126 massive
($M_{\star}>10^{10}M_{\sun}$) GV galaxies at $0.5<z<2.5$ and split it into the
higher overdensity quarter and the lower overdensity quarter. The results
shows that GV galaxies in denser environment have higher $n$ values and lower
sSFR at $0.5<z<1$, while there is no discernible distinction at $1<z<2.5$. No
significant enlarging or shrinking is found for GV galaxies in different
environments within the same redshift bin. It suggests that a dense
environment would promote the growth of bulge and suppress star formation
activity of GV galaxies at $0.5<z<1.5$, but would not affect the galaxy size.
We also study the dependence of the fraction of three populations (Blue Cloud,
Green Valley, and Red Sequence) on both environments and $M_{\star}$. At a
given $M_{\star}$, blue cloud fraction goes down with increasing environment
density, while red sequence fraction is opposite. For the most massive GV
galaxies, a sharp drop appears in the denser environment. Coupled with the
mass dependence of three fractions in different redshift bins, our result
implies that stellar mass and environments jointly promote the quenching
process. Such dual effect is also confirmed by re-calculating the new
effective GV fraction as the number of GV galaxies over the number of non-
quiescent galaxies.

Green valley galaxies (683); Galaxy environments (2029); Star formation(1569);
Galaxy quenching(2040); Stellar properties (1624)

##  1 Introduction

A flood of observations on large samples of galaxies found evident cessation
of star formation activities during the evolution of galaxies. The cessation
of star formation is widely known as “quenching” and is assumed to produce
passive galaxies in which the star formation rate (SFR) is very low, commonly
resulting in the bimodal distribution of galaxies. In the color–magnitude
diagram, the narrow red peak, in which abundant quiescent galaxies (QGs) and a
small amount of dusty star-forming galaxies (SFGs) are distributed along with
a linear sequence (Blanton & Moustakas 2009), is normally called “red
sequence” (RS). The blue peak occupying an extended region mainly consists of
SFGs (Kauffmann et al. 2003), similarly called “blue cloud” (BC). An
intermediate zone between BC and RS is commonly known as “green valley” (GV),
initially proposed and described in several GALEX papers (e.g., Martin et al.
2007; Salim et al. 2007; Wyder et al. 2007).

GV galaxies (see the review of Salim 2014) are often thought to be in a
transitional phase. The existence of GV galaxies argues for a continuum of
properties from SFGs to QGs (Wyder et al. 2007; Salim 2014). Over the past
decades, mechanisms of galaxy quenching remains an unsolved problem. Many
previous researches emphasized the role of stellar mass ($M_{\star}$) and
environment in the cessation of star formation, generally referred to as “mass
quenching” (Peng et al. 2010b; Somerville & Davé 2015; Penny et al. 2018) and
“environment quenching” (Peng et al. 2010b; Darvish et al. 2015). A plausible
explanation of “mass quenching” is that the feedback from active galactic
nucleus (AGN) plays an important role in regulating star formation and in
quenching galaxies (Hopkins et al. 2005, 2014; Somerville & Davé 2015; Penny
et al. 2018). At the same time, a number of authors have considered that the
external environment might be the key factor of suppressing the star formation
via some mechanisms to consume the gas reservoir, such as major/minor merger
(Springel et al. 2005), ram pressure stripping (Gunn & Gott 1972), and
harassment (Farouki & Shapiro 1981). High environmental density might
subsequently increase dust attenuation (e.g., Koyama et al. 2013; Sobral et
al. 2016) and enrich interstellar medium of SFGs (Sobral et al. 2015; Darvish
et al. 2015).

It is noteworthy that galaxy morphology varies with the environment.
Generally, galaxies in the lower density environments (field) are bluer, more
star-forming, and more disc-like, while galaxies in higher density
environments (cluster) are older, redder, less star-forming, and more
elliptical (Dressler 1984; Kauffmann et al. 2004). Dressler (1980) found that
there is a definite relationship between local galaxy density and
morphological type at $z<0.06$, which is further confirmed in other studies
(Guzzo et al. 1997; Goto et al. 2003; Fasano et al. 2015). The dependence of
galaxy morphology on the environment appears not only in the local Universe
but also at intermediate and high redshifts ($z\sim 1-2$, e.g., Dressler et
al. 1997; van der Wel et al. 2007; Allen et al. 2015, 2016). Allen et al.
(2015) compared the morphology of SFGs and QGs in different environments
(cluster vs. field). They found that cluster SFGs had higher Sérsic indices
($n$) than field SFGs at $0<z<2$, while there is no difference in light
profile and galaxy size for QGs between field and cluster environments.
However, it is not consistent with the results of others that QGs in the
cluster have shallower profiles (lower $n$) and larger sizes (Bassett et al.
2013; Strazzullo et al. 2013; Yoon et al. 2017).

Considering that the environment affects both star formation activity and
morphology in galaxies at the transitional stage, GV galaxies could be one of
the suitable samples to study the environmental effect on physical properties.
In our previous series of work, Gu et al. (2018) have assembled a large sample
of massive ($M_{\star}>10^{10}~{}M_{\odot}$) RS, GV, and BC galaxies at
$0.5\leqslant z\leqslant 2.5$ in five fields of 3D-HST/CANDELS, and
investigated their morphology, dust content, and environments and revealed a
mass-dependent “downsizing” quenching picture. Then, Gu et al. (2019) analyzed
the mass dependence of morphology and star formation activity for three
populations. We found that the structural properties of GV galaxies are
intermediate between those of BC and RS galaxies at fixed stellar mass bins at
$z<2$, and both GV and BC galaxies have similar sizes and compactness at the
high-mass end. It implies that GV galaxies could go through a morphological
transformation of bulge buildup at $z<1.5$, which is consistent with our
results of Lu et al. (2021). We found the effect of the morphological
quenching mechanism on star formation activity at $0.5<z<2.5$, after
eliminating any possible AGN candidates and considering the stellar mass
influence. But we had not studied the effect of the environment on the
physical properties of GV galaxies until Gu et al. (2021) defined a
dimensionless overdensity (1+$\delta^{{}^{\prime}}$) as the environmental
indicator by adopting a Bayesian method to consider the contributions of all
the $N$th nearest neighbors. Based on this improved environmental method, in
this work we set out to explore empirical relations between galaxy
environments and other physical properties, including parametric and non-
parametric structure, and star-forming parameters, for GV galaxies at
$0.5<z<2.5$. To understand the quenching process for massive galaxies, we also
analyze the transformation of different fractions for RS, GV, and BC galaxies.

The structure of our paper is organized as follows. Data and sample selection
are described in Section 2. We present the environmental dependence of
structural parameters and sSFR in Section 3. Section 4 contains our analysis
about the different fractions for RS, GV, and BC populations and the
distribution of effective fractions for GV galaxies. Finally, a summary is
given in Section 5. Throughout our paper, we adopt the cosmological parameters
as following: $H_{0}=70\,{\rm km~{}s}^{-1}\,{\rm Mpc}^{-1}$,
$\rm\Omega_{m}=0.30$, and $\Omega_{\Lambda}=0.70$. All magnitudes adopted in
this paper are in the AB system.

##  2 Data and Sample selection

###  2.1 3D-HST and CANDELS

The 3D-HST and CANDELS programs cover over $\sim$900 arcmin2 in five different
fields: AEGIS, COSMOS, GOODS-N, GOODS-S, and UDS, observed by a number of
space-based and some ground-based telescopes. It thus provides abundant
imaging data from ultraviolet (UV) to infrared (IR) bands via the high-quality
WFC3 and ACS spectroscopy and photometry (Grogin et al., 2011; Koekemoer et
al., 2011; Skelton et al., 2014; Momcheva et al., 2016). In the
3D-HST/CANDELS, many physical properties of galaxies have been measured by
utilizing the homogeneous multi-band data, including the parameters of the
stellar population (Skelton et al. 2014; Whitaker et al. 2014; Momcheva et al.
2016) and structure (van der Wel et al. 2014).

In this work, redshifts and rest-frame colors are taken from Momcheva et al.
(2016). It is an updated version of the photometric catalog from Skelton et
al. (2014) with the grism redshifts from the fits of G141 grism spectroscopy.
We refer to Momcheva et al. (2016) for the full details. Following Wang et al.
(2017), stellar mass and dust attenuation are re-estimated based on the
stellar population synthesis models from Maraston (2005) via the FAST code
(Kriek et al., 2009). The Maraston (2005) models are known to give a better
description of stellar populations for high-redshift SFGs whereby taking the
contribution of the thermally pulsating asymptotic giant branch stars into
account. We assume an exponentially declining star formation history with an
e-folding time $\tau\sim 0.1-10$ Gyr, a Kroupa (2001) initial mass function
(IMF) and solar metallicity. The dust attenuation ($A_{V}$) varies from 0 to 4
in steps of 0.1 following the Calzetti et al. (2000) law. It is worthy of
mentioning that we prefer to take the spectroscopic (or grism) redshifts if
available. Otherwise, we will use the photometric redshifts from Skelton et
al. (2014) instead.

Figure 1: The extinction-corrected rest-frame U-V color of GV galaxies as a
function of stellar mass in three redshift bins. In each panel, the background
grayscale shallows represent the distribution for all our parent massive
galaxies. Green points are GV galaxies. The two criteria for each redshift bin
from Wang et al. (2017) are separately shown as red and blue bands, where the
edge of the ribbon corresponds to the range of redshift. The number of GV
galaxies in each redshift bin has been shown in the bottom of each panel.

###  2.2 Structural Parameters

Quantitative morphological and structural analysis has been developed and
complemented in the past decades. There are two main measurements of
morphology: parametric modelling of surface brightness profile (Sersic 1968;
Peng et al. 2002, 2010a) and non-parametric morphology (Abraham et al. 1994;
Lotz et al. 2004a; Conselice 2014).

Parametric models are important in describing light profile to estimate the
galaxy size and bulge-to-disk decomposition (Buitrago et al. 2008; Wuyts et
al. 2011; Lang et al. 2014; van der Wel et al. 2014). Sérsic indices ($n$) and
effective radius ($r_{\rm e}$) in CANDELS fields are taken from van der Wel et
al. (2014) which have been measured using GALFIT (Peng et al. 2002). 3D-HST
catalog has been matched with J band and H band of CANDELS by Rainbow
Database111https://arcoirix.cab.inta-csic.es/Rainbow_Database/Home.html. The
rest-frame optical morphologies are traced by J-band (F125W) imaging at
$0.5<z<1.5$ and by H-band (F160W) imaging at $1.5<z<2.5$ in this work.

Non-parametric methods are usually used to identify irregular structures in
galaxies, especially for high-redshift galaxies, with high irregularity (Bluck
et al. 2012; Conselice 2014). We have performed the measurements of the Gini
coefficient ($G$; Lotz et al. 2004a) and the second-order moment of the 20%
brightest pixels ($M_{20}$; Lotz et al. 2004a) on the NIR images using the
Morpheus software developed by Abraham et al. (2007). More detailed
definitions and calculations for these non-parametric measurements refer to Gu
et al. (2018). The effect of signal-to-noise ratio (S/N) on the measurement of
non-parametric parameters has been tested by previous work (Kong et al., 2009;
Wang et al., 2012; Fang et al., 2015; Gu et al., 2018, 2019; Lu et al., 2021).
In sample selection (see Section 2.5) we adopt a flag of $\tt use\\_phot=1$
that ensures a reliable detection in $H$-band (F160W) with S/N$>$3 and thus
reliable non-parametric measurements.

###  2.3 Star Formation Rates

Young and massive stars produce a large amount of UV photons, which are partly
absorbed by interstellar dust and re-emit in IR. Thus, accurate SFR estimation
should include both unobscured and obscured parts. The unobscured SFR can be
derived from the observed rest-frame UV luminosity, while the obscured one can
be estimated through a single mid-infrared measurement, e.g., MIPS 24 $\mu$m
(Battisti et al., 2015; Lin et al., 2016).

In this work, we employ the total SFR by combining UV and IR emissions,
provided by Whitaker et al. (2014). Considering the Bell et al. (2005)
conversion and the Kroupa (2001) IMF, the $\rm SFR_{UV+IR}$ can be estimated
by:

${\rm SFR_{UV+IR}}[M_{\sun}~{}{\rm yr}^{-1}]=9.8\times 10^{-11}(L_{\rm
IR}+2.2L_{\rm UV})/L_{\sun},$ (1)

where $L_{\rm UV}=1.5\times L_{\rm 2800}$ is estimated by the rest-frame
continuum luminosity at 2800Å, and $L_{\rm IR}$ is the integrated luminosity
at 8$\sim$1000 $\mu$m converted from the Spitzer/MIPS 24 $\mu$m data using a
single luminosity-independent template (Franx et al. 2008; Wuyts et al. 2008).

If the MIPS 24 $\mu$m data are unavailable, we assume the dust attenuation
curve from Calzetti et al. (2000) and correct the effect of dust attenuation
on $\rm SFR_{UV}$. In this case, the corrected $\rm SFR_{UV}$ can be derived
as follows:

${\rm SFR_{UV,corr}}[M_{\sun}~{}{\rm yr}^{-1}]={\rm SFR_{UV}}\times
10^{0.4\times 1.8\times A_{V}},$ (2)

where ${\rm SFR_{UV}}=3.24\times 10^{-10}\times L_{2800}/L_{\sun}$. The factor
of 1.8 converts $A_{V}$ to that at 2800Å when adopting the Calzetti et al.
(2000) attenuation curve.

###  2.4 Measurements of Environmental Overdensities

The different definitions of environmental density describe disparate physical
properties on different physical scales (Muldrew et al. 2012; Etherington &
Thomas 2015). In the paper, we adopt the dimensionless overdensity as the
indicator of the relative local environment of galaxies, described in detail
in Gu et al. (2021). A magnitude-limited sample is selected for the
measurements of environmental overdensities. This sample mixes galaxies with
spectroscopic, grism, or photometric redshifts and is selected by the
following criteria: (1) H-band apparent magnitudes $\rm F160W<25$ that
guarantees the uncertainty of photometric redshifts $\sigma_{z}=0.02$, (2) a
redshift range of $0.5<z<2.5$, and (3) flag $\tt use\\_phot=1$ that ensures
reliable photometries of galaxies (see Section 2.5 for more details). About
30% of galaxies in this sample with spectroscopic or grism redshifts provide a
more convinced derivation of galaxies environments, although using purely
photometric redshifts to construct the overdensities would not change our
results in general. In the following, we summarise the relevant information on
the environmental indicator.

The Bayesian metric in Cowan & Ivezić (2008) is adopted to estimate the local
environmental density, which is defined as $\Sigma^{{}^{\prime}}_{N}$
$\propto$ 1/($\Sigma^{N}_{i=1}d_{i}^{2}$). Here $d_{i}$ is the projected
distance of the $i$th nearest neighbor in the projected two-dimensional space
within the individual redshift slice. This Bayesian environmental density
considers the contribution of all $N$th nearest neighbors, which can improve
accuracy in mapping the probability density distribution compared to the
traditional method (Ivezić et al. 2005). Given that photometric redshifts have
large uncertainties, we choose the redshift slice as $\left|\Delta
z\right|=2\sigma_{z}(1+z)$, where $\sigma_{z}=0.02$ is the typical uncertainty
of photometric redshifts of our sample, and $z$ is the redshift of target
galaxy to estimate the environment.

With increasing redshift, the comoving number densities of the observed
galaxies are expected to decrease. Thus, we adopt a dimensionless overdensity
of Gu et al. (2021), 1+$\delta^{{}^{\prime}}_{N}$, as the indicator of galaxy
environment:

$1+\delta^{{}^{\prime}}_{N}=\frac{\Sigma^{{}^{\prime}}_{N}}{\langle\Sigma^{{}^{\prime}}_{N}\rangle_{\rm
uniform}}=\frac{\Sigma^{{}^{\prime}}_{N}}{k^{{}^{\prime}}_{N}\Sigma_{\rm
surface}},$ (3)

where $\langle\Sigma^{{}^{\prime}}_{N}\rangle_{\rm uniform}$ is the standard
value of Bayesian density when galaxies are distributed in the uniform
environment. At a given density $\Sigma_{\rm surface}$,
$\langle\Sigma^{{}^{\prime}}_{N}\rangle_{\rm uniform}$ can be calculated by a
linear correction coefficient $k^{{}^{\prime}}_{N}$ (see also in the Appendix
of Gu et al. 2021). Thus, $\log(1+\delta^{{}^{\prime}}_{N})>$ 0 indicates that
the environmental density of a galaxy exceeds the density standard in the
uniform condition and vice versa.

Due to the Poisson noise and the possible contamination of foreground and
background galaxies, a small value of $N$ may cause fluctuation in the density
values. We therefore adopt the overdensity based on the distances to all 10
nearest neighbors ($\Sigma^{{}^{\prime}}_{10}$) as the indicator of galaxy
environments in this work. statistically

###  2.5 Sample Selection

Based on the magnitude-limit sample, we focus on the massive galaxies with
$\log(M_{\star}/M_{\sun})\geqslant 10$. The flag of $\tt use\\_phot=1$ is set
to choose galaxies with a reliable detection following these criteria: 1) not
a star, or bright enough to be a reliable galaxy; 2) not close to a bright
star; 3) well-exposed in the F125W and F160W bands; 4) a S/N of f$\\_$F160W /
e$\\_$F160W $>$ 3; 5) has a passable photometric redshift fit; 6) has a “non-
catastrophic” stellar population fit, with $\log M_{\star}>$ 0 (Skelton et
al., 2014). To ensure reliable measurement of the structural parameters, we
also make an additional cut on H-band, H${}_{\rm F160W}<$ 24.5, which could
get rid of faint galaxies and guarantee much higher reliable photometric
redshifts (Momcheva et al. 2016). Finally, the parent sample contains 7850
massive galaxies at 0.5 $<$ z $<$ 2.5 from five 3D-HST/CANDELS fields, about
58% of galaxies have spectroscopic or grism redshifts. Considering about half
of sample with photometric redshifts, the bias from galaxies spectroscopic
redshifts which trend to be brighter might be relieved.

Given that the intrinsic rest-frame colors depend on stellar mass and
redshift, we use the extinction-corrected rest-frame U-V colors to divide
galaxies into RS, GV, and BC populations as well, as Wang et al. (2017) did.
The separation criteria are as follows:

$\displaystyle(U-V)_{\rm rest}-\Delta
A_{V}=0.126\log(M_{\star}/M_{\sun})+0.58-0.286z;\,\,$ $\displaystyle(U-V)_{\rm
rest}-\Delta A_{V}=0.126\log(M_{\star}/M_{\sun})-0.24-0.136z,\,\,$

where $\Delta A_{V}=0.47\times A_{V}$ is the extinction correction of rest-
frame U-V color, the value of 0.47 is the correction factor from the Calzetti
et al. (2000) attenuation law. Then we obtain 2566 RS galaxies, 2126 GV
galaxies, and 3158 BC galaxies.

In Figure 1, we show the extinction-corrected rest-frame U-V color of GV
galaxies as a function of stellar mass in three redshift bins from 0.5 to 2.5.
The criteria for each redshift bin from Wang et al. (2017) are separately
shown as red and blue bands, where the width of the ribbon corresponds to the
range of redshift. GVs are distributed below the upper limit of the first
separation and above the lower limit of the second separation. The background
grayscales represent the distributions for all massive galaxies in our parent
sample, and also show the separation for BC and RS. The number of GV in each
bin is also shown in each panel. Due to the redshift-dependence of the GV
definition, we can find that high-redshift galaxies tend to be bluer than low-
redshift galaxies in Figure 1.

##  3 Environmental Effect on Physical Properties

In this section, we discuss the effect of the environment on different
physical properties of GV galaxies, including parametric ($n$ and re), non-
parametric ($G$ and M20) structures, and sSFR. Considering that the dominated
quenching mechanism might vary across cosmic time (Iovino et al. 2010), we
carry out the research in three redshift bins (0.5 $\leq z<$ 1.0, 1.0 $\leq
z<$ 1.5, and 1.5$\leq z\leq$ 2.5). In each redshift bin, GV galaxies are
divided into four equal bins according to the local overdensity
1+$\delta^{{}^{\prime}}_{\rm 10}$. In the following, we compare the physical
properties of GV galaxies in the highest and lowest overdensity bins, namely
the highest and lowest density quarters. To qualify the correlation between
stellar mass and different physical properties and the corresponding
differences between the highest and lowest density quarters in three redshift
bins, we calculate the Spearman correlation coefficients and perform
2-Dimensional Kolmogorov–Smirnov (2D-KS) test (Peacock 1983; Fasano &
Franceschini 1987), respectively. It is assumed that there is a significant
difference between two density quarters when the p-value of 2D-KS test is
smaller than 0.05. All results are listed in Table 2.

Figure 2: Sérsic index $n$ (top panels) and effective radius re (bottom
panels) as functions of $M_{\star}$ in the highest (dark green) and lowest
(light green) density quarters. The error bars are the 2$\sigma$ uncertainties
drawn from the bootstrap method with the 1000 times resamples.

Figure 3: Gini coefficient $G$ (top panels) and the second order 20% bright
pixels $M_{20}$ (bottom panels) as functions of $M_{\star}$ in two extreme
environments. The colors and markers are same as Figure 2.

###  3.1 Parametric Structures

For a galaxy fitted with a single Sérsic profile, the disc-like galaxies have
a characteristic of $n\sim$ 1, whereas the Sérsic indices of bulge-dominated
galaxies are considered to be $n>$ 2.5 (Shen et al. 2003; Cebrián & Trujillo
2014). Figure 2 shows the dependence of Sérsic index $n$ and effective radius
re of GV galaxies on stellar mass in two extreme environments. Dark green
squares represent the corresponding median values for GV galaxies in the
highest density quarter, while light green squares represent those in the
lowest density quarter. The error bars represent the 2$\sigma$ uncertainties
drawn from the bootstrap method with the 1000 times resamples.

In the top panels of Figure 2, we find that the difference in $n$ between the
highest and lowest environments is significant at $0.5<z<1$ with the p-value
of 2D-KS test $\sim$ 0.003, which reveals that Sérsic indices ($n$) of GV
galaxies are larger in denser environments. It is unlikely to see that GV
galaxies are disc-like in the densest environments at $0.5<z<1$, regardless of
stellar mass. There are no clear distinctions between two opposite
environments at higher redshift ($z>1$) with the corresponding 2D-KS p-value
$\gg$ 0.05. Our result indicates that galaxy environments have a significant
impact on galaxy structure at $0.5<z<1$. This difference in Sérsic indices
between two opposite environments is also reported by Paulino-Afonso et al.
(2019), which found that there is an environmental dependence of Sérsic
indices ($n$) in different stellar mass bins when considering the entire
sample including both SFGs and QGs, with denser environments having galaxies
with higher Sérsic indices ($n$) at $z\sim 0.84$. In addition, there is a
correlation between $n$ and redshift: the overall indices $n$ of high-redshift
galaxies are smaller than these of low-redshift galaxies. In our sample, it is
easier to observe more disc-like GV galaxies at high-redshift.

The bottom panels of Figure 2 show the dependence of galaxy size on the
environment density and stellar mass. The galaxy size increases by a small
amount with stellar mass in given redshift bins, with the Spearman
coefficients $>$ 0.3, regardless of the environments. This is consistent with
the expectation about the general mass–size relation: more massive galaxies
tend to be characterized by a larger radius than their less massive
counterparts on average (Shen et al. 2003; van der Wel et al. 2014). Our
result provides a specific mass–size relation for massive GV galaxies at
$0.5<z<2.5$ in two extreme environments. We find no significant difference in
size between galaxies in the highest density and lowest density for three
redshift bins with KS p-value $\gg$ 0.05, suggesting that a denser environment
is not effective enough for inhibiting or stimulating galaxy size.

Figure 4: Dependence of sSFR for massive GV galaxies on the environment and
stellar mass at $0.5<z<2.5$. The colors and markers are the same as Figure 2.

###  3.2 Non-parametric Structures

We show the $G$ and $M_{20}$ distributions as functions of stellar mass in
different redshift and environment bins in Figure 3. For massive GV galaxies,
the average $G$ increases with the cosmic time, while M20 generally decreases
with decreasing redshifts. A large $G$ value hints that the light distribution
of a galaxy tends to be not uniform but concentrated. This could explain why
$n$ has a similar redshift evolution as shown in Figure 2. Peth et al. (2016)
have approved that the $G$ is sensitive to the $n$. Different from $G$, the
decrease of $M_{20}$ is associated with the disappearance of sub-structures in
galaxies, such as bars, spiral arms, and bright cores (Lotz et al., 2004b). In
other words, these non-parametric measurements of massive GV galaxies show
substantial variations with redshift, i.e, these galaxies at higher redshift
tend to have more uniform light distribution globally but more sub-structures
locally. The results of median (or KS) test between the highest and lowest
environments show that p-values for $G$ in all redshift bins are larger than
0.05, indicating no environment-related variation. Figure 3 also shows the
dependence of $M_{20}$ on both stellar mass and at environment overdensity
quantiles at $0.5<z<1.0$, with 2D-KS p-value $<$ 0.05. The corresponding
Spearman coefficients for higher and lower density quarters are about -0.476
and -0.497, respectively, suggesting that $M_{20}$ is negatively dependent on
stellar mass. GV galaxies with higher mass tend to have less prominent
substrctures.

Figure 5: The fraction of galaxies in different stellar mass bins as the
function of overdensity and redshift for BC (top), GV (middle), and RS
(bottom). The dotted line roughly divides the environment into low, medium,
and high overdensity. For each panel, colors from light to dark indicate a
redshift from low to high for different populations. At a fixed stellar mass
and environment, the sum of the fractions of all three populations equals one.

###  3.3 Specific Star Formation Rate

In Figure 4, we show the sSFR vs. $M_{\star}$ distributions in two extreme
environments in three redshift bins. The negative Spearman correlation
coefficients ($<$ -0.1) shown in Table 2 reveal that there is a steady drop of
sSFR with increasing stellar mass at $0.5<z<1$ and $1<z<1.5$. It is consistent
with results of Bauer et al. (2005), which reported that sSFR decreases with
increasing galaxy stellar mass at $0<z<1.5$. It shows the mass dependence for
massive GV galaxies, which decreases at high redshift.

In each redshift bin, the dark green squares represent the sSFR–$M_{\star}$
relation in the highest density quarter, while the light ones are in the
lowest density quarter. At $1.0<z<2.5$, there is no significant difference of
sSFR for GV galaxies in two opposite environments, with the corresponding
2D-KS p-value $>$ 0.05. While at 0.5 $\leq z<1$, it could be clearly seen the
difference of sSFR between the highest and lowest environmental density, where
p-value $\sim$ 0.003, especially for the most massive GV galaxies: a sharp
drop at the high-mass end for GV galaxies in the highest overdensity. A
possible explanation for this sharp decline is that a denser environment would
suppress the star formation activities for more massive GV galaxies at lower
redshift. It hints the effect of “environment quenching”. We would discuss the
quiescent fraction in Section 4 to explain the quenching process more.

Table 1: Outline of CAE Architecture and Layer Configuration.  | $0.5<z<1.0$ |  | $1.0<z<1.5$ |  | $1.5<z<2.5$
---|---|---|---|---|---
| Spearman | Spearman | 2D KS |  | Spearman | Spearman | 2D KS |  | Spearman | Spearman | 2D KS
| (high) | (low) |  |  | (high) | (low) |  |  | (high) | (low) |
$n$ | 0.292 | 0.244 | 0.003 |  | 0.056 | 0.084 | 0.567 |  | -0.046 | -0.031 | 0.528
re | 0.498 | 0.584 | 0.417 |  | 0.429 | 0.447 | 0.274 |  | 0.373 | 0.322 | 0.266
$G$ | 0.109 | 0.174 | 0.123 |  | 0.155 | 0.087 | 0.528 |  | 0.049 | 0.071 | 0.251
M20 | -0.476 | -0.497 | 0.015 |  | -0.214 | -0.155 | 0.407 |  | -0.137 | -0.081 | 0.755
sSFR | -0.258 | -0.275 | 0.003 |  | -0.327 | -0.167 | 0.260 |  | -0.085 | -0.181 | 0.192
Table 2: Spearman correlation coefficients and 2-D KS test for physical
properties between different environment at $0.5<z<1.0$, $1.0<z<1.5$, and
$1.5<z<2.5$.

##  4 Discussion

During the sample selection in Section 2.5, we have divided galaxies into
three different populations (BC, GV, and RS) according to the corrected rest-
frame U-V color. The fraction of each population could give an insight into
the quenching timescale, which is defined as the number of each population
over the number of the whole sample, i.e., $f_{\rm i}=N_{\rm i}/(N_{\rm
BC}+N_{\rm GV}+N_{\rm RS})$, i = BC, GV, and RS. If the quenching timescale is
large, BC galaxies will pass through the GV phase slowly and then become
quiescent, resulting in a large $f_{\rm GV}$. Otherwise, BC galaxies go
through the transition and become quiescent quickly, without a significant
rise of $f_{\rm GV}$ (Jian et al. 2020). Therefore, comparing the
distributions of $f_{\rm GV}$ as a function of environments with those of
$f_{\rm BC}$ and $f_{\rm RS}$ might help us understand the quenching process.

Figure 6: The effective fraction of GV galaxies in different stellar mass bins
as the function of overdensity and redshift. We define the new effective
fraction by normalizing the number of GV galaxies with the number of non-
quiescent galaxies.

###  4.1 Fractions of Three Populations

Figure 5 shows the fractions of galaxies belonging to BC (top), GV (middle),
and RS (bottom) populations as functions of overdensity in different stellar
mass and redshift bins. The vertical dot-dashed lines separate the environment
into low ($\log(1+\delta^{{}^{\prime}}_{10})<0.1$), medium
($0.1<\log(1+\delta^{{}^{\prime}}_{10})<0.6$), and high overdensity
($\log(1+\delta^{{}^{\prime}}_{10})>0.6$) from the left to right. Regardless
of redshifts, $f_{\rm BC}$ keeps a steady decline with the rise of local
density, while it is the opposite for the RS fraction, which decreases
gradually.

At the low-mass end ($10.0<\log M_{\star}/M_{\odot}<10.4$), $f_{\rm BC}$ and
$f_{\rm RS}$ are much more dependent on the overdensity compared to $f_{\rm
GV}$. While at the high-mass end ($\log M_{\star}/M_{\odot}>$ 10.8), it
becomes that $f_{\rm GV}$ and $f_{\rm RS}$ are both sensitive to the
overdensity, while $f_{\rm RS}$ is relatively stable. The comparison of
$f_{\rm GV}$ in the high- and low-mass bins implies that the quenching
timescale of the massive GV galaxies is larger than that of the less massive
GV galaxies, because more massive GV galaxies are undergoing the quenching
processes, especially in the field at the high-mass end. So it can reveal the
real fact of the influence of stellar mass on the galaxy quenching, called the
“mass quenching”.

In addition, when we consider the effects of both the environment and stellar
mass, it is expected that more massive galaxies in the high overdensity are
easier to be quenched into the RS population, resulting in a higher $f_{\rm
RS}$ in the denser environment at the high-mass end as shown in the panels f)
and i) of Figure 5. What is interesting in Figure 5 is the general pattern of
decline in the GV fraction with the increasing overdensity at $0.5<z<1$, even
for the less massive galaxies. Meanwhile, galaxies in denser environments have
a higher $f_{\rm RS}$, regardless of $M_{\star}$ and redshifts, which may
imply that denser environments have additional effects on the acceleration of
quenching. In summary, we find that GV and BC galaxies would be more likely to
transform into RS galaxies in the denser environments for all the massive
galaxies. Our results also display the effect of stellar mass on the quenching
process as previous researches (Iovino et al. 2010; Peng et al. 2010b), thus
suggesting that the joint role of stellar mass and environment promotes the
quenching process, as evidence of “mass quenching” and “environment quenching”
(Vulcani et al. 2012; Darvish et al. 2017).

###  4.2 Effective Fraction of GV Galaxies

In general, BC galaxies could be considered as the progenitors of GV galaxies.
With the cosmic time, the abundance of RS galaxies and the shortage of BC
galaxies naturally leads to the drop-down of GV population in Figure 5,
especially for galaxies at the high-mass end. Recently, Jian et al. (2020)
have pointed out that the contamination of RS galaxies might bias the relative
fraction of GV galaxies. Therefore, the effective fraction, defined as the
relative number of GV galaxies over the number of non-quiescent galaxies
(i.e., BC and GV galaxies), $f_{\rm eff,GV}=N_{\rm GV}/(N_{\rm BC}+N_{\rm
GV})$, can be a better indicator to reflect the galaxy transitional phase more
realistically, after removing the effect due to the dominance of the quiescent
galaxies.

Based on the new normalized fraction, Figure 6 shows $f_{\rm eff,GV}$ as a
function of overdensity in bins of redshift and stellar mass. After
eliminating the RS galaxies, $f_{\rm eff,GV}$ at $0.5<z<1$ in the low
overdensity environment is lower than those in the environments with higher
overdensity , and the relations between GV fraction and environment reverse
compared to those shown in Figure 5. This result suggests that the observed
negative correlations between $f_{\rm GV}$ and overdensity are mostly due to
the reduction of the non-quiescent galaxies with increasing environment
densities. The steady increasing trend of $f_{\rm eff,GV}$ with
$\log(1+\delta^{\prime}_{10})$ in the lower redshift bin might support a
scenario that environment indeed boosts the beginning of quenching process for
SFGs in intermediate-redshift Universe. At $1<z<2.5$, there is no slow upward
trend of $f_{\rm eff,GV}$ with the increase of local overdensity, implying
that environment has no impact on the beginning of the quenching, although it
plays an important role in the complete cessation of the star formation
activities as seen in the bottom panels of Figure 5.

On the other hand, the overall of $f_{\rm eff,GV}$ at fixed redshift and
$\log(1+\delta^{\prime}_{10})$ significantly increases with $M_{\star}$, which
reveals a strong mass dependence of $f_{\rm eff,GV}$. For more massive GV
galaxies at $1<z<1.5$, a sharp drop of the effective fraction of GV from
middle to high overdensity is observed, which might be the consequence of the
combination of effects from both stellar mass and environment, which is the
same as the explains in Section 4.1.

##  5 Summary

By utilizing the multi-wavelength data from five 3D-HST/CANDELS fields, we
construct a sample of 7850 massive ($M_{\star}>10^{10}M_{\sun}$) galaxies at
$0.5<z<2.5$. Based on the extinction-corrected rest-frame U-V color, we
separate the parent sample into BC, GV, and RS galaxies (Gu et al. 2018,
2019), resulting in a total number of 2126 GV galaxies. We provide empirical
relations for GV galaxies between environment and different physical
properties, including $n$, $r_{\rm e}$, $G$, $M_{20}$, and sSFR. We also
analyze the fractions of three populations as functions of environments and
redshift. Our conclusions are summarized as follows:

1\. At $0.5<z<1$, GV galaxies have larger Sérsic indices ($n$) in the denser
environment. The environment seems to have a significant impact on $n$ at
$0.5<z<1$. It is unlikely to observe disc-like GV galaxies in the densest
environment, which suggests that a denser environment would promote the growth
of bulge of low-redshift GV galaxies. We find no significant difference in
galaxy size (re) between galaxies in environments with the highest and the
lowest overdensity at fixed redshift, indicating that a denser environment is
not effective enough to influence galaxy size.

2\. Non-parametric measurements of massive GV galaxies show substantial
variations with redshift that GV galaxies are more disc-like at higher
redshift. Neither stellar mass nor environments have significant impact on $G$
in different redshift bins, while there is a dependence of $M_{20}$ on both
stellar mass and environment at $0.5<z<1$.

3\. At $0.5<z<1.0$, there is a decrease of sSFR from the lowest to the highest
environmental density, especially at the high-mass end. A plausible
explanation is that a denser environment would suppress the star formation
activity of GV galaxies, especially for massive GV galaxies at low redshift.

4\. We also discuss the effect of the environment on the fraction for three
different populations (BC, GV, RS). At the low-mass end (10.0$<\log
M_{\star}/M_{\odot}<$10.4), the BC and RS fractions are much more dependent on
the overdensity compared to the GV fraction. While at the high-mass end ($\log
M_{\star}/M_{\odot}>$ 10.8), it becomes that the GV and RS fractions are
sensitive to the overdensity, and the BC fraction is relative stable. It
implies that the quenching timescale of the massive GV galaxies is longer than
that of the less massive GV galaxies, which reveals the influence of “mass
quenching”. There is a general pattern of decline in the GV fraction and
increase in the RS fraction with the increase of environment density at
$0.5<z<1$. It suggests that denser environments have effects on the quenching
process. Considering the effect of mass on three fractions, it could be
explained that both denser environments and mass aggregation might promote the
galaxy quenching process.

5\. The $f_{\rm eff,GV}$ rises gradually with the increase of environmental
density, suggesting that environment boosts the beginning of the quenching
process. The overall $f_{\rm eff,GV}$ is found to have a positive correlation
with $M_{\star}$, regardless of redshift and environments. Both original
fraction and new effective fraction suggest that stellar mass and environments
jointly promote the quenching process.

This work is based on observations taken by the 3D-HST Treasury Program (GO
12177 and 12328) with the NASA/ESA HST, which is operated by the Association
of Universities for Research in Astronomy, Inc., under NASA contract
NAS5-26555. This work is supported by the Strategic Priority Research Program
of Chinese Academy of Sciences (No. XDB 41000000), the National Key R&D
Program of China (2017YFA0402600), the NSFC grant (No. 11973038), and the
China Manned Space Project with No. CMS-CSST-2021-A07. Z.S.L acknowledges the
support from China Postdoctoral Science Foundation (2021M700137). Y.Z.G
acknowledges support from China Postdoctoral Science Foundation funded project
(2020M681281). G.W.F. acknowledges support from Yunnan Applied Basic Research
Projects (2019FB007).

## References

  * Abraham et al. (1994) Abraham, R. G., Valdes, F., Yee, H. K. C., & van den Bergh, S. 1994, ApJ, 432, 75
  * Abraham et al. (2007) Abraham, R. G., Nair, P., McCarthy, P. J., et al. 2007, ApJ, 669, 184
  * Allen et al. (2016) Allen, R. J., Kacprzak, G. G., Glazebrook, K., et al. 2016, ApJ, 826, 60
  * Allen et al. (2015) Allen, R. J., Kacprzak, G. G., Spitler, L. R., et al. 2015, ApJ, 806, 3
  * Bassett et al. (2013) Bassett, R., Papovich, C., Lotz, J. M., et al. 2013, ApJ, 770, 58
  * Battisti et al. (2015) Battisti, A. J., Calzetti, D., Johnson, B. D., & Elbaz, D. 2015, ApJ, 800, 143
  * Bauer et al. (2005) Bauer, A. E., Drory, N., Hill, G. J., & Feulner, G. 2005, ApJ, 621, L89
  * Bell et al. (2005) Bell, E. F., Papovich, C., Wolf, C., et al. 2005, ApJ, 625, 23
  * Blanton & Moustakas (2009) Blanton, M. R., & Moustakas, J. 2009, ARA&A, 47, 159
  * Bluck et al. (2012) Bluck, A. F. L., Conselice, C. J., Buitrago, F., et al. 2012, ApJ, 747, 34
  * Buitrago et al. (2008) Buitrago, F., Trujillo, I., Conselice, C. J., et al. 2008, ApJ, 687, L61
  * Calzetti et al. (2000) Calzetti, D., Armus, L., Bohlin, R. C., et al. 2000, ApJ, 533, 682
  * Cebrián & Trujillo (2014) Cebrián, M., & Trujillo, I. 2014, MNRAS, 444, 682
  * Conselice (2014) Conselice, C. J. 2014, ARA&A, 52, 291
  * Cowan & Ivezić (2008) Cowan, N. B., & Ivezić, Ž. 2008, ApJ, 674, L13
  * Darvish et al. (2017) Darvish, B., Mobasher, B., Martin, D. C., et al. 2017, ApJ, 837, 16
  * Darvish et al. (2015) Darvish, B., Mobasher, B., Sobral, D., et al. 2015, ApJ, 814, 84
  * Dressler (1980) Dressler, A. 1980, ApJ, 236, 351
  * Dressler (1984) —. 1984, ARA&A, 22, 185
  * Dressler et al. (1997) Dressler, A., Oemler, Augustus, J., Couch, W. J., et al. 1997, ApJ, 490, 577
  * Etherington & Thomas (2015) Etherington, J., & Thomas, D. 2015, MNRAS, 451, 660
  * Fang et al. (2015) Fang, G., Ma, Z., Kong, X., & Fan, L. 2015, ApJ, 807, 139
  * Farouki & Shapiro (1981) Farouki, R., & Shapiro, S. L. 1981, ApJ, 243, 32
  * Fasano & Franceschini (1987) Fasano, G., & Franceschini, A. 1987, MNRAS, 225, 155
  * Fasano et al. (2015) Fasano, G., Poggianti, B. M., Bettoni, D., et al. 2015, MNRAS, 449, 3927
  * Franx et al. (2008) Franx, M., van Dokkum, P. G., Förster Schreiber, N. M., et al. 2008, ApJ, 688, 770
  * Goto et al. (2003) Goto, T., Yamauchi, C., Fujita, Y., et al. 2003, MNRAS, 346, 601
  * Grogin et al. (2011) Grogin, N. A., Kocevski, D. D., Faber, S. M., et al. 2011, ApJS, 197, 35
  * Gu et al. (2018) Gu, Y., Fang, G., Yuan, Q., Cai, Z., & Wang, T. 2018, ApJ, 855, 10
  * Gu et al. (2019) Gu, Y., Fang, G., Yuan, Q., et al. 2019, ApJ, 884, 172
  * Gu et al. (2021) Gu, Y., Fang, G., Yuan, Q., Lu, S., & Liu, S. 2021, ApJ, 921, 60
  * Gunn & Gott (1972) Gunn, J. E., & Gott, J. Richard, I. 1972, ApJ, 176, 1
  * Guzzo et al. (1997) Guzzo, L., Strauss, M. A., Fisher, K. B., Giovanelli, R., & Haynes, M. P. 1997, ApJ, 489, 37
  * Hopkins et al. (2005) Hopkins, P. F., Hernquist, L., Cox, T. J., et al. 2005, ApJ, 630, 705
  * Hopkins et al. (2014) Hopkins, P. F., Kereš, D., Oñorbe, J., et al. 2014, MNRAS, 445, 581
  * Iovino et al. (2010) Iovino, A., Cucciati, O., Scodeggio, M., et al. 2010, A&A, 509, A40
  * Ivezić et al. (2005) Ivezić, Ž., Vivas, A. K., Lupton, R. H., & Zinn, R. 2005, AJ, 129, 1096
  * Jian et al. (2020) Jian, H.-Y., Lin, L., Koyama, Y., et al. 2020, ApJ, 894, 125
  * Kauffmann et al. (2004) Kauffmann, G., White, S. D. M., Heckman, T. M., et al. 2004, MNRAS, 353, 713
  * Kauffmann et al. (2003) Kauffmann, G., Heckman, T. M., White, S. D. M., et al. 2003, MNRAS, 341, 54
  * Koekemoer et al. (2011) Koekemoer, A. M., Faber, S. M., Ferguson, H. C., et al. 2011, ApJS, 197, 36
  * Kong et al. (2009) Kong, X., Fang, G., Arimoto, N., & Wang, M. 2009, ApJ, 702, 1458
  * Koyama et al. (2013) Koyama, Y., Smail, I., Kurk, J., et al. 2013, MNRAS, 434, 423
  * Kriek et al. (2009) Kriek, M., van Dokkum, P. G., Labbé, I., et al. 2009, ApJ, 700, 221
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231
  * Lang et al. (2014) Lang, P., Wuyts, S., Somerville, R. S., et al. 2014, ApJ, 788, 11
  * Lin et al. (2016) Lin, Z., Fang, G., & Kong, X. 2016, AJ, 152, 191
  * Lotz et al. (2004a) Lotz, J. M., Primack, J., & Madau, P. 2004a, AJ, 128, 163
  * Lotz et al. (2004b) —. 2004b, AJ, 128, 163
  * Lu et al. (2021) Lu, S., Fang, G., Gu, Y., et al. 2021, ApJ, 913, 81
  * Maraston (2005) Maraston, C. 2005, MNRAS, 362, 799
  * Martin et al. (2007) Martin, D. C., Small, T., Schiminovich, D., et al. 2007, ApJS, 173, 415
  * Momcheva et al. (2016) Momcheva, I. G., Brammer, G. B., van Dokkum, P. G., et al. 2016, ApJS, 225, 27
  * Muldrew et al. (2012) Muldrew, S. I., Croton, D. J., Skibba, R. A., et al. 2012, MNRAS, 419, 2670
  * Paulino-Afonso et al. (2019) Paulino-Afonso, A., Sobral, D., Darvish, B., et al. 2019, A&A, 630, A57
  * Peacock (1983) Peacock, J. A. 1983, MNRAS, 202, 615
  * Peng et al. (2002) Peng, C. Y., Ho, L. C., Impey, C. D., & Rix, H.-W. 2002, AJ, 124, 266
  * Peng et al. (2010a) —. 2010a, AJ, 139, 2097
  * Peng et al. (2010b) Peng, Y.-j., Lilly, S. J., Kovač, K., et al. 2010b, ApJ, 721, 193
  * Penny et al. (2018) Penny, S. J., Masters, K. L., Smethurst, R., et al. 2018, MNRAS, 476, 979
  * Peth et al. (2016) Peth, M. A., Lotz, J. M., Freeman, P. E., et al. 2016, MNRAS, 458, 963
  * Salim (2014) Salim, S. 2014, Serbian Astronomical Journal, 189, 1
  * Salim et al. (2007) Salim, S., Rich, R. M., Charlot, S., et al. 2007, ApJS, 173, 267
  * Sersic (1968) Sersic, J. L. 1968, Atlas de Galaxias Australes
  * Shen et al. (2003) Shen, S., Mo, H. J., White, S. D. M., et al. 2003, MNRAS, 343, 978
  * Skelton et al. (2014) Skelton, R. E., Whitaker, K. E., Momcheva, I. G., et al. 2014, ApJS, 214, 24
  * Sobral et al. (2015) Sobral, D., Stroe, A., Dawson, W. A., et al. 2015, MNRAS, 450, 630
  * Sobral et al. (2016) Sobral, D., Stroe, A., Koyama, Y., et al. 2016, MNRAS, 458, 3443
  * Somerville & Davé (2015) Somerville, R. S., & Davé, R. 2015, ARA&A, 53, 51
  * Springel et al. (2005) Springel, V., Di Matteo, T., & Hernquist, L. 2005, ApJ, 620, L79
  * Strazzullo et al. (2013) Strazzullo, V., Gobat, R., Daddi, E., et al. 2013, ApJ, 772, 118
  * van der Wel et al. (2007) van der Wel, A., Holden, B. P., Franx, M., et al. 2007, ApJ, 670, 206
  * van der Wel et al. (2014) van der Wel, A., Franx, M., van Dokkum, P. G., et al. 2014, ApJ, 788, 28
  * Vulcani et al. (2012) Vulcani, B., Poggianti, B. M., Fasano, G., et al. 2012, MNRAS, 420, 1481
  * Wang et al. (2012) Wang, T., Huang, J.-S., Faber, S. M., et al. 2012, ApJ, 752, 134
  * Wang et al. (2017) Wang, T., Elbaz, D., Alexander, D. M., et al. 2017, A&A, 601, A63
  * Whitaker et al. (2014) Whitaker, K. E., Franx, M., Leja, J., et al. 2014, ApJ, 795, 104
  * Wuyts et al. (2008) Wuyts, S., Labbé, I., Förster Schreiber, N. M., et al. 2008, ApJ, 682, 985
  * Wuyts et al. (2011) Wuyts, S., Förster Schreiber, N. M., Lutz, D., et al. 2011, ApJ, 738, 106
  * Wyder et al. (2007) Wyder, T. K., Martin, D. C., Schiminovich, D., et al. 2007, ApJS, 173, 293
  * Yoon et al. (2017) Yoon, Y., Im, M., & Kim, J.-W. 2017, ApJ, 834, 73


# Progressive Neural Image Compression
with Nested Quantization and Latent Ordering

###### Abstract

We present PLONQ, a progressive neural image compression scheme which pushes
the boundary of variable bitrate compression by allowing quality scalable
coding with a single bitstream. In contrast to existing learned variable
bitrate solutions which produce separate bitstreams for each quality, it
enables easier rate-control and requires less storage. Leveraging the latent
scaling based variable bitrate solution, we introduce nested quantization, a
method that defines multiple quantization levels with nested quantization
grids, and progressively refines all latents from the coarsest to the finest
quantization level. To achieve finer progressiveness in between any two
quantization levels, latent elements are incrementally refined with an
importance ordering defined in the rate-distortion sense. To the best of our
knowledge, PLONQ is the first learning-based progressive image coding scheme
and it outperforms SPIHT, a well-known wavelet-based progressive image codec.

††∗ Equal contribution††1 Qualcomm AI Research is an initiative of Qualcomm
Technologies, Inc.††2 Work completed during internship at Qualcomm
Technologies Inc.

###### keywords:

progressive coding, quality scalable coding, embedded coding, variable bitrate
compression, nested quantization

##  1 Introduction

Recent developments of learning based lossy image [1, 2, 3, 4, 5, 6] and video
compression [7, 8, 9, 10, 11] schemes have witnessed remarkable success in
achieving state-of-the-art rate-distortion (R-D) performance compared to
traditional codecs in a variety of benchmark datasets. Leveraging the
expressiveness of deep auto-encoder networks, these works compress an image by
learning a nonlinear transform between the image and the latent, along with a
learned entropy model in order to encode the quantized latent. The
optimization objective is often framed as a tradeoff between the Rate (R), the
amount of bits used to encode the source, and the Distortion (D), the distance
between the input source and the reconstructed source:

$\min_{\theta}\mathbb{E}_{x}\left[D_{\theta}(x)+\beta R_{\theta}(x)\right]$

where $\beta$ is the tradeoff parameter and $\theta$ represents the model
parameters. During training, $\theta$ is typically optimized from end-to-end
using stochastic gradient descent.

Fig. 1:  Qualitative comparison3 of PLONQ and BPG444. For PLONQ (top),
reconstructions at different qualities can be obtained by truncating a single
bitstream, whereas for BPG444 (bottom), separate bitstreams have to be stored
one for each bitrate option.  33footnotetext: Displayed image is a cropped
version of 00012_TE_1512x2016.png from JPEG AI testset [12]. Reported in this
figure is the PNSR and bitrate of the full-size reconstruction.

Most of these learning based methods, however, are not readily suitable for
large scale deployment because in most of learning based codecs [1, 2, 4, 5,
6], separate models need to be trained to accommodate different bitrate
targets. To address this issue, Choi et al. [13] proposed a conditional auto-
encoder to make the parameters in the encoder and decoder network be dependent
on the rate distortion trade-off parameter, $\beta$ so that a single model can
adapt to different rate-distortion tradeoffs. More recently, [14, 15, 3]
proposed latent scaling based variable bitrate compression which is
essentially learning to adjust the quantization step size of the latents.
After training, the interpolation of the learned scaling parameters can lead
to continuous variable bitrate performance. Other attempts [16, 17] have also
been made to address the variable bitrate targets.

While variable bitrate solutions make learning based image compression more
practical for deployment, they require storing multiple copies of the latent
code used for different bitrates, i.e., quality levels [14, 15]. In contrast,
with progressive coding, the transmitted bitstream is completely embedded,
which be truncated at various points and reconstructed into a series of lower
bitrate images [18]. Therefore it significantly reduces storage and simplifies
rate control.

Existing literature [19] allowing for progressiveness adopted an encoder
parameterized by a recurrent neural network, so that it can incrementally
transmit the quantized latents. However, the inference time is slow for
recurrent neural networks and their performance is not competitive with BPG or
the family of hyperprior based solutions [1, 6]. In this study, we present
PLONQ, a Progressive coding based neural image compression method via Latent
Ordering and Nested Quantization. PLONQ is built upon the latent scaling based
solution while being amenable for deployment. In summary, our work has the
following main contributions:

  1. 1.

We demonstrate the effectiveness of a simple latent scaling based variable
bitrate solution using a pretrained high bitrate model without any retraining,
dubbed as Naive Scaling.

  2. 2.

We propose nested quantization which enables progressive coding across
different quantization levels.

  3. 3.

We find sorting latent variables element-wise by prior standard deviation
works well to obtain better truncation points between two quantization levels.

Combining nested quantization with latent ordering, PLONQ achieves better
performance than SPIHT, a well-known progressive coding algorithm, and
performs on par with BPG at high bitrate.

##  2 Latent Scaling

We base our method on the mean-scale hyperprior model [2] and inherit the
notations therein: we denote the input image by $x$, and the analysis
transform (encoder) and synthesis transform (decoder) by $g_{a}$ and $g_{s}$
respectively. Both $g_{a}$ and $g_{s}$ are parameterized by convolutional
neural networks. The output of the encoder is the continuous latent space
tensor $y=g_{a}(x)$, which has a Gaussian prior with its mean $\mu$ and
standard derivation $\sigma$ predicted based on hyper latent. We use
$P(\cdot)$ to denote the probability value on an interval and use
$\mathbb{P}(\cdot)$ to denote the discretized probability value.

The idea of latent scaling [14, 15], is to apply a scaling factor $s$ to the
latent $y$. Because the quantization step size is fixed, this leads to a
different tradeoff between rate and distortion. For ease of exposition we
consider a single latent variable, so that both $y$ and $s$ are scalar. The
diagram of the latent scaling is depicted in Fig 2. The latent $y$ is scaled
with $1/s$, where we restrict $s>1$. In the quantization step (shown in the
blue shaded rectangle in Fig 2, we choose to center the scaled latent $y/s$ by
its prior mean $\mu/s$, apply the rounding operator $\lfloor\cdot\rceil$ on
$(y-\mu)/s$ (so that the estimated mean $\mu$ learned by the hyper-encoder is
on the grid), and then add the offset $\mu/s$ back. The dequantized latent
$y(s)$ is obtained by multiplying $s$ after the quantization block, i.e.

$\displaystyle\centering y(s)\triangleq\@add@centering$
$\displaystyle\left\lfloor\frac{y-\mu}{s}\right\rceil s+\mu.$

The prior probability of $y(s)$ used in entropy coding can be derived from the
original prior density by change of variables:

$\displaystyle\mathbb{P}_{Y(s)}(y(s))=\mathbb{P}_{\frac{Y(s)}{s}}\left(\frac{y(s)}{s}\right)=\mathbb{P}_{\frac{Y(s)}{s}}\left(\left\lfloor\frac{y-\mu}{s}\right\rceil+\frac{\mu}{s}\right)$
$\displaystyle=\int^{\lfloor(y-\mu)/s\rceil+\frac{1}{2}}_{\lfloor(y-\mu)/s\rceil-\frac{1}{2}}p_{(y-\mu)/s}(u)du=\int^{y^{+}(s)}_{y^{-}(s)}p_{y}(v)dv,$
(1)

where $y^{+}(s)\triangleq y(s)+\frac{s}{2}$ and $y^{-}(s)\triangleq
y(s)-\frac{s}{2}$ can be interpreted as the upper and lower boundary of the
effective quantization bin.

Note in Eq. (1), we first apply the change of variable formula for discrete
random variables in the first line, and then apply a change of variable
formula for continuous random variables from the second to the third line.
According to Eq (1), applying the scaling $s$ is equivalent to changing the
quantization bin width from 1 to $s$. Hence increasing the latent scaling
value $s$ will lead to larger quantization bin width and hence smaller bitrate
in the entropy coding, as well as larger distortion of the reconstructed
image.

Fig. 2: Mean-scale hyperprior with latent scaling. EC is entropy encoder, ED
is entropy decoder. Note that scaling $s$ is applied to latent $y$ after being
passed to hyper codec so that the prior parameters $\mu,\sigma$ can be shared
by $y(s)$ for different $s$.

##  3 Nested Quantization

In this section, we introduce nested quantization, one of the key techniques
we use to construct a progressive bitstream. Let us start with a pre-trained
mean-scale hyperprior model and consider $K$ quantization levels defined by a
set of $K$ latent scaling factors $\\{s_{1},s_{2},...,s_{K}\\}$ with
$s_{1}<s_{2}<...<s_{K}$. We define a scheme, where for each level, the same
scaling factor is applied across all latent elements. Effectively, these $K$
quantization bin sizes translate into $K$ different bitrate options. We name
this simple single-model variable-bitrate scheme as _Naive Scaling_. Despite
its simplicity, In Section 5 we show that it achieves performance comparable
to a more involved scheme [14] that uses $K$ sets of learnable per-channel
scaling factors optimized for $K$ Lagrange multipliers $\beta$.

Naturally, the information contained in the discrete latents quantized with
different scaling factors follows a total ordering – the one quantized with
smaller bins is always a refinement of the larger. Progressiveness across
different quantization levels can then be achieved if we can find a way to
embed the information of coarse level quantization into its finer counterpart.
Indeed, a principled way to achieve this progressiveness is to encode a finer
quantized latent with probabilities conditioned on its coarser quantized
values. We term this scheme _nested quantization_ and detail it next.

In nested quantization, the encoding happens in $K$ stages. In the first
stage, we quantize $y$ with the coarsest bin $s_{K}$ and encode it with
$\mathbb{P}\left(y(s_{K})\right)$ as defined in Eq (1). In the remaining
stages, we iteratively refine the quantization of $y$ using bin width
$s_{k-1}$ that is one level finer than $s_{k}$, for $1\leq k<K$. In
particular, we encode an extra piece of information of $y$ quantized to a
finer grained level through the conditional probability below,

$\displaystyle\mathbb{P}\left(y(s_{k})|y(s_{i})_{i>k}\right)=$ $\displaystyle
P(I_{k})/P(I_{k+1})\text{ for }1\leq k<K,$

where $P(I)\triangleq\int_{v\in I}p_{y}(v)dv$, and $I_{k}$ is defined below as
iterative intersections of quantization bins:

$\displaystyle I_{K}$
$\displaystyle\triangleq\left[y^{-}(s_{K}),y^{+}(s_{K})\right],\text{ and }$
$\displaystyle I_{k}$ $\displaystyle\triangleq
I_{k+1}\cap\left[y^{-}(s_{k}),y^{+}(s_{k})\right]\text{ for }1\leq k<K.$

An illustration of this procedure with $K=2$ can be found in Fig 3. The code
generated by these $K$ stages form a bitstream that embeds $K$ different
bitrates, and the total length of the bitstream is111Here we assume a perfect
entropy coder.

Fig. 3: Illustration of nested quantization of $y$ when $K=2$.

$\displaystyle\underbrace{\log\frac{1}{\mathbb{P}\left(y\left(s_{K}\right)\right)}}_{\begin{subarray}{c}\text{codeword
length w.r.t.}\\\ \text{coarsest quant. level
$s_{K}$}\end{subarray}}+\sum_{1\leq
k<K}\underbrace{\log\frac{1}{\mathbb{P}\left(y\left(s_{k}\right)|y\left(s_{i}\right)_{i>k}\right)}}_{\begin{subarray}{c}\text{codeword
length w.r.t. refined}\\\ \text{information from }s_{k+1}\text{ to
}s_{k}\end{subarray}}=\log\frac{1}{P(I_{1})}.$

There are two issues of this scheme as can be seen from the above equation:
(1) conditional probability computation requires tracking of intersected
quantization boundaries, which adds to implementation complexity; (2) the sum
of codeword length $\log 1/P(I_{1})$ is in general larger222
$P\left(I_{1}\right)=P\left(\left[y^{-}\left(s_{1}\right),y^{+}\left(s_{1}\right)\right]\cap
I_{2}\right)\leq
P\left(\left[y^{-}\left(s_{1}\right),y^{+}\left(s_{1}\right)\right]\right)=\mathbb{P}\left(y\left(s_{1}\right)\right)$
than the codeword length with respect to the finest quantization level $\log
1/\mathbb{P}\left(y(s_{1})\right)$.

These two issues can be resolved if we consider a set of _fully nested
quantization levels_ 333This can be viewed as a generalized version of bit-
plane coding. where the set of grid points of the coarser quantization bin is
always a subset of the finer. In other words, we can define quantization
levels in such a way that
$I_{k}=I_{k+1}\cap\left[y^{-}(s_{k}),y^{+}(s_{k})\right]=\left[y^{-}(s_{k}),y^{+}(s_{k})\right]$,
which simplifies the above equation as

$\displaystyle\log\frac{1}{\mathbb{P}\left(y\left(s_{K}\right)\right)}+\sum_{1\leq
k<K}\log\frac{\mathbb{P}\left(y\left(s_{k+1}\right)\right)}{\mathbb{P}\left(y\left(s_{k}\right)\right)}=\log\frac{1}{\mathbb{P}\left(y\left(s_{1}\right)\right)}.$

With this we are able to preserve the performance of the highest bit-rate
model at the end of the progressive coding procedure.

In practice, we find it beneficial to further generalize this idea to
quantization levels with uneven grid, which is applied to PLONQ and explained
in Section 5.

##  4 Latent Ordering

To obtain more truncation points in the embedded bitstream between two
quantization levels, latent variables can be refined incrementally instead of
all at once. The problem is how to find the optimal order to refine them, so
that wherever the resulting bitstream is truncated, the reconstruction has the
highest possible quality. The embedding principle states the latent that
reduces the distortion the most per bit should be coded first [20].

Depending on the embedding granularity, the latent tensor $y$ is divided into
coding units $\\{y_{1},...,y_{N}\\}$. All the latent elements in one coding
unit are refined together at once, corresponding to one truncation point. For
the latent tensor with shape $(C,H,W)$444$C,H,W$ stands for the number of
channels (feature maps), height and width of each channel., we have considered
three types of coding unit for the latent: (1) one channel, (2) one pixel, and
(3) one element, which corresponds to a latent slice of size $(1,H,W)$,
$(C,1,1)$ and $(1,1,1)$ respectively.

Given an order $\rho=(\rho_{1},...,\rho_{N})$, ordered coding units
$y_{\rho}=(y_{\rho_{1}},...,y_{\rho_{N}})$ are refined one by one from scaling
$s_{k}$ to $s_{k-1}$. In mean-scale hyperprior model, the priors of latent
elements are independent conditioned on the hyper latent. Thus the bitrate
increase of refining $y_{\rho_{t}}$, i.e. $\Delta
R(y_{\rho_{t}})=-\log\mathbb{P}\left(y_{\rho_{t}}(s_{k-1})|y_{\rho_{t}}(s_{k})\right)$
can be calculated in parallel once the hyper latent is decoded and it is
independent of $\rho$. However, with a nonlinear ConvNet decoder, the
reduction in distortion depends on the all other ordered coding units, i.e.
$\Delta
D\left(y_{\rho_{t}}|y_{\rho}\right)=D\left(y_{\rho}(t-1)\right)-D\left(y_{\rho}(t)\right)$,
where $y_{\rho}(t)=(y_{\rho_{\leq t}}(s_{k-1}),y_{\rho_{>t}}(s_{k}))$, and
$D(y)=\text{MSE}(x,g_{s}(y))$ is the distortion for latent $y$. Thus refining
ordered latent by $\rho$ leads to a set of R-D points
$\mathcal{H}(\rho)=\\{\sum_{i\leq t}\Delta R(y_{\rho_{i}}),\sum_{i\leq
t}\Delta D(y_{\rho_{i}}|y_{\rho})\\}_{t=1}^{N}$. The optimal order $\rho^{*}$
is the one under which the convex hull of $\mathcal{H}(\rho^{*})$ is better
than that of other orders in the Pareto optimal sense [21].

Following embedding principle, $\frac{\Delta D}{\Delta R}$ should be sorted in
descending order. We use a greedy procedure to calculate $\frac{\Delta
D}{\Delta R}$ under an initial order, e.g. the index order, as the first
sorting criterion. The second criterion is the bitrate $\Delta R$ of a coding
unit, because we observe sorting latent channels by $\Delta R$ and by
$\frac{\Delta D}{\Delta R}$ lead to similar R-D curve. Since the standard
derivation $\sigma$ of the latent prior is correlated to the expected bitrate,
it is considered as the third criterion. Note that the first two criteria
incur bitrate overhead for coding the order, while the last one does not since
$\sigma$ is already known when decoding $y$.

Fig. 4: Coding units are incrementally refined between two quantization levels
($s_{2}=10,000$, i.e. $y(s_{2})=\mu$, $s_{1}=1$) according to a latent order
determined by a sorting criterion.

(a) Quantization levels and breakdown of components in the bitstream.

(b) Rate-distortion comparison on
JPEG AI testset[12]

(c) BD-rate savings relative to BPG444 on
JPEG AI testset[12]

Fig. 5: Illustration of PLONQ, and performance comparison of PLONQ and
baseline schemes described in Table 1.

We experimented with 6 combinations of coding unit and sorting criterion, and
find the one with the best R-D performance. The latent ordering results on the
JPEG AI testset [12] is shown in Fig 4. It is natural to choose channels as
the coding unit because each channel is usually considered as a learned
feature map in ConvNet codec. Indeed channel ordering has been studied in
ordered representation learning with nested dropout of latent channels [22]
and progressive decoding with a channel-wise auto-regressive prior model
(Section $G$ in the appendix of [1]). With channel-wise ordering, the more
principled sorting criterion $\frac{\Delta D}{\Delta R}$ performs slightly
better than the alternatives while being much more expensive to calculate.
When using a single element as coding unit, $\frac{\Delta D}{\Delta R}$
becomes too expensive to calculate, but sorting latent elements by $\sigma$
performs much better than sorting channels by any criterion, and does not
require to transmit the latent ordering, thus sorting element units by
$\sigma$ is chosen for latent ordering in PLONQ.

##  5 Experiment

In this section, we show experimental results of Naive Scaling, a plug-and-
play variable bitrate solution and PLONQ, our proposed progressive coding
scheme, compared against a set of baseline solutions listed in Table 1.

For PLONQ, we use four quantization levels as the basis for nested
quantization, and a mean-scale hyperprior ([2] without spatial autoregresssive
context) trained with $\beta=1e-4$ as the base model. We enforce the
quantization grid to be fully nested in the sense that a coarser quantization
grid is always aligned with a finer one. In practice, we find it beneficial to
adopt an uneven quantization grid design, where the center bin size can be
different from the other bin sizes. Through experiments we identify the set of
quantization levels illustrated in Fig 5(a) as a good design choice555these
four quantization levels form a nested set of deadzone quantizers [23].

To favor fine-grained rate-control, PLONQ computes an image-specific ordering
$\rho$ of latent elements and then incrementally updates the latent with finer
quantization level element by element following the order. Based on the
analysis in Section 4, we adopt the order computed by sorting the per-element
standard deviation $\sigma$ from the output of the hyper decoder. An
illustration of breakdown of the bitstream can be found in lower half of Fig
5(a). Note that the use of image-specific latent element ordering incurs no
extra bit overhead, since the ordering can be derived after hypercode is
obtained.

Table 1: Baseline image compression schemes Label |  Description
---|---
MeanScale |  Mean-Scale hyperprior model ([2] without spatial autoregresssive context). Four models trained separately with $\beta$=1e-4, 3e-4, 1e-3, and 3e-3 for 2 million steps.
BPG444 |  Better Portable Graphics[24] (HEVC All-intra)
QSF |  Quality Scaling Factor[14]. 8 sets of learnable per-channel latent scaling factors trained from $\beta=$1e-4 to 5e-2 on a pre-trained and freezed MeanScale($\beta$=1e-4) model.
NaiveScaling |  Constant latent scaling factors $s=1,2,\ldots,8$ applied to a pre-trained MeanScale($\beta$=1e-4) model.
ND |  MeanScale($\beta$=1e-4) with Nested Dropout[22] training.
SPIHT |  Set Partitioning In Hierarchical Trees[18]

###  5.1 Performance evaluation

We evaluate the performance of PLONQ against an array of baseline schemes
outlined in Table 1, which divides into three categories (as delimited in the
table): (1) multi-model compression scheme (2) non-progressive single model
variable bitrate scheme (3) progressive compression scheme. Among them, BPG444
and SPIHT are traditional codecs and the rest are learning-based codecs.

Fig 5(b) and 5(c) show the RD curve (bpp vs PSNR) as well as the BD-rate[25]
saving relative to BPG444 computed on JPEG AI dataset [12]. The markers on the
progressive solution RD curves indicate the achievable rate options. As shown
in the two figures, PLONQ is significantly better than the learned nested
dropout [22]. Further PLONQ outperforms the well-known conventional
progressive coding scheme SPIHT [18] by a considerable margin uniformly across
the whole bpp range, and it is even competitive to the non-progressive coding
scheme BPG444 at high bpp regime. The same observation is made on Kodak and
Tecnick dataset, for which the results can be found in the appendix.

##  6 Conclusion

We have introduced PLONQ, a learning based progressive image codec using
nested quantization and latent ordering. It uniformly outperforms the existing
wavelet-based progressive image codec SPIHT and matches or even outperforms
BPG444 in the high bitrate region. The effectiveness of PLONQ proves itself to
be a solid baseline for learning based progressive codec and gives the first
glimpse of its potential to enable progressive neural video coding.

## References

  * [1] David Minnen and Saurabh Singh,  “Channel-wise Autoregressive Entropy Models for Learned Image Compression,”  in IEEE International Conference on Image Processing (ICIP), 2020\.
  * [2] Johannes Ballé David Minnen and George D. Toderici,  “Joint Autoregressive and Hierarchical Priors for Learned Image Compression ,”  in Advances in Neural Information Processing Systems 31 (NeurIPS), 2018.
  * [3] Tong Chen, Haojie Liu, Zhan Ma, Qiu Shen, Xun Cao, and Yao Wang,  “Neural Image Compression via Non-Local Attention Optimization and Improved Context Modeling,” 2019.
  * [4] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto,  “Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [5] Johannes Ballé, Valero Laparra, and Eero P. Simoncelli,  “End-to-End Optimized Image Compression,”  in International Conference on Learning Representations (ICLR), 2017\.
  * [6] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston,  “Variational Image Compression with a Scale Hyperprior,”  in International Conference on Learning Representations (ICLR), 2018\.
  * [7] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici,  “Scale-space flow for end-to-end optimized video compression,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [8] Ruihan Yang, Yibo Yang, Joseph Marino, and Stephan Mandt,  “Hierarchical Autoregressive Modeling for Neural Video Compression,”  in International Conference on Learning Representations (ICLR), 2021\.
  * [9] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao,  “DVC: An End-To-End Deep Video Compression Framework,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [10] Amirhossein Habibian, Ties van Rozendaal, Jakub M. Tomczak, and Taco S. Cohen,  “Video compression with rate-distortion autoencoders,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [11] Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere, and Taco S. Cohen,  “Feedback recurrent autoencoder for video compression,”  in Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020.
  * [12] JPEG AI Testset, 2020,  https://jpegai.github.io/test_images/.
  * [13] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee,  “Variable Rate Deep Image Compression With a Conditional Autoencoder,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [14] T. Chen and Z. Ma,  “Variable Bitrate Image Compression with Quality Scaling Factors,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
  * [15] Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, and Yihui Feng,  “G-VAE: A Continuously Variable Rate Deep Image Compression Framework,”  2020\.
  * [16] Tiansheng Guo, Jing Wang, Ze Cui, Yihui Feng, Yunying Ge, and Bo Bai,  “Variable Rate Image Compression With Content Adaptive Optimization,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [17] Jing Zhou, Akira Nakagawa, Keizo Kato, Sihan Wen, Kimihiko Kazui, and Zhiming Tan,  “Variable Rate Image Compression Method With Dead-Zone Quantizer,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [18] A. Said and W. A. Pearlman,  “A New, Fast, and Efficient Image Codec Based on Set Partitioning in Hierarchical Trees,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 6, no. 3, pp. 243–250, 1996.
  * [19] George Toderici, Sean M. O’Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar,  “Variable Rate Image Compression with Recurrent Neural Networks,”  in International Conference on Learning Representations (ICML), 2016\.
  * [20] David Taubman, Erik Ordentlich, Marcelo Weinberger, and Gadiel Seroussi,  “Embedded block coding in JPEG 2000,”  Signal Processing: Image Communication, vol. 17, no. 1, pp. 49–72, 2002.
  * [21] David Taubman,  “High Performance Scalable Image Compression with EBCOT,”  IEEE Transactions on image processing, vol. 9, no. 7, pp. 1158–1170, 2000.
  * [22] Oren Rippel, Michael Gelbart, and Ryan Adams,  “Learning Ordered Representations with Nested Dropout,”  in International Conference on Machine Learning, 2014, pp. 1746–1754.
  * [23] G. J. Sullivan,  “Efficient scalar quantization of exponential and Laplacian random variables,”  IEEE Transactions on Information Theory, vol. 42, no. 5, pp. 1365–1374, 1996.
  * [24] Fabrice Bellard,  Better Portable Graphics, 2014,  https://bellard.org/bpg/.
  * [25] Gisle Bjontegaard,  “Calculation of Average PSNR Differences Between RD-curves,”  VCEG-M33, 2001.

##  Appendix A Dataset preprocessing

The following two image in the original JPEG AI test dataset incur more GPU
memory than is available to us for mean-scale hyper-prior inference, and thus
we cropped it to 3680x2160 (cropped from top) for all our experiments.

00014_TE_3680x2456.png

00015_TE_3680x2456.png

##  Appendix B Evaluation of BPG444 and SPIHT

BPG444 [24] results are generated with the following command:

bpgenc -e x265 -q [qp] -f 444

-o [output file] [input file]

The results on Kodak dataset (shown in 6) is validated against that provided
in
https://github.com/tensorflow/compression/
blob/master/results/image_compression/
kodak/PSNR_sRGB_RGB/bpg444.txt

SPIHT[18] results are obtained by encoding each image with a target bpp of 2.0
and decoding by truncating the bitstream by a step size of 0.1bpp. The curves
are generated by averaging PSNR across images for each of the fixed bpps.

##  Appendix C Additional experiment results on Kodak and Tecnick testset

Fig 6 and 7 show the performance comparison of PLONQ and baseline schemes
described in Table 1 on Kodak; Fig 8 and 9 shows the results on Tecnick test
dataset (100 images with 1200 x 1200 resolution).

Fig. 6: Rate-distortion comparison on Kodak. Fig. 7: BD-rate savings relative
to BPG444 on Kodak. Fig. 8: Rate-distortion comparison on Tecnick testset.
Fig. 9: BD-rate savings relative to BPG444 on Tecnick testset.