# Gas accretion onto galaxies and Kelvin-Helmholtz turbulence

Itzhak Goldman1,2 and Robert Fleck3
1 Department of Physics, Afeka College, Tel Aviv, Israel
2 Department of Astrophysics, Tel Aviv University, Tel Aviv, Israel
3 Department of Physical Sciences, Embry-Riddle Aeronautical University,
Daytona Beach, Florida, USA<EMAIL_ADDRESS>
###### Abstract

Continued star formation over the lifetime of a galaxy suggests that low
metalicity gas is steadily flowing in from the circumgalactic medium. Also,
cosmological simulations of large-scale structure formation imply that gas is
accreted onto galaxies from the halo inside which they formed. Direct
observations are difficult, but in recent years observational indications of
gas inflows from a circumgalactic medium were obtained. Here we suggest an
indirect observational probe: looking for large-scale (exceeding few kpc)
turbulence caused by the accretion. As a specific example we consider an
accretion flow coplanar with the galaxy disk, and argue that Kelvin-Helmholtz
turbulence will be generated. We employ a semi-analytic model of turbulence
and derive the expected turbulence power spectrum. The latter turns out to be
of a distinctive shape that can be compared with observational power spectra.
As an illustrative example we use parameters of the Milky Way galaxy.

###### keywords:

galaxies-star formation-turbulence-ism

††pubyear: 2022††pagerange: Gas accretion onto galaxies and Kelvin-Helmholtz
turbulence–References

##  1 Introduction

The observational evidence that star formation goes on continuously in
galaxies suggests that there is fresh gas supply from the circumgalactic
medium (CGM); (see e.g. Elmegreen (2016); Bland-Hawthorn et al. (2017).
Simulations of cosmological large-scale structure formation support this
picture; e.g. (Kereš et al., 2005). For a recent review on the physics,
astrophysics and observational status of the CGM see Tumlinson, Peeples, &
Werk (2017).

Direct observations are difficult; nevertheless, in recent years observation
were made suggesting the existence of such an accretion (Elmegreen, Struck, &
Hunter, 2014; Elmegreen et al., 2016; Kacprzak, 2017; Vulcani et al., 2018;
Martin et al., 2019; Zabl et al., 2019; Das et al., 2020; Luo et al., 2021;
Ianjamasimanana et al., 2022).

Recently, observations carried out for disk galaxies at a redshift $z\sim 0.2$
(Ho et al., 2017; Ho, Martin, & Turner, 2019; Martin et al., 2019; Ho &
Martin, 2020) and at redshift $z\sim 1$ (Zabl et al., 2019) indicate the
existence of cold gas inflowing coplanar with the disk in the near CGM (30–80
kpc) corotating with the disk. The estimated inflow radial velocities are in
the range of (20–60) km/s. A very recent simulation by Trapp et al. (2022)
conducted for a MIlky Way (MW)-type galaxy obtained results consistent with
these observations, finding that the deviation of accreting gas from the
circular velocity of the galaxy disk gas is quite small and that there is a
radial inflow velocity $\sim 40\ $ km/s.

In the present paper we examine a possible observational signature of disk-
plane accretion of CGM gas. We find that such an accretion could generate
Kelvin-Helmholtz (KH) turbulence at the disk outskirts. Specifically, we
address the case in which the inflowing gas is cool and rotates in the same
direction as that of the galactic disk. as supported by the observations and
simulations; the parameters of the MW galaxy are used as an illustrative
example.

The resulting turbulence is large scale ($\sim 10\ $ kpc) with a distinctive
velocity power spectrum. There is a range of wavenumbers in which the power
spectrum has a logarithmic slope of $\sim-1.1$ which is quiet distinct from
the Kolmogorov slope of $-5/3$ or $-2$ corresponding to compressible
turbulence. In order to test the sensitivity of the results to values of the
adopted parameters, we repeat the computation with others that are consistent
with the observations. While the largest scale of the turbulence and the
turbulent velocity do depend on the adopted values of the parameters used, the
power spectrum retains its particular shape.

##  2 Kelvin-Helmholtz Instability

The KH instability (KHI) arises when two fluids with different velocities and
(usually) densities form an interface perpendicular to a gravitational field
(Chandrasekhar, 1961). It is manifested in the case of wind flowing over a
body of water, and also in various astrophysical settings (Fleck, 1983, 1984,
1989; Gómez & Ostriker, 2005; Mandelker et al., 2016; Fleck, 2020). In this
paper we apply the linear growth rate of the instability in a semi-analytic
model of turbulence (Canuto, Goldman, & Mazzitelli, 1996). The latter re-
normalizes the growth rate self-consistently taking into account non-linear
turbulence. The turbulence model provides the spatial power spectrum of the
turbulent velocity field.

The linear growth rate of the KHI is given by Chandrasekhar (1961)

$\displaystyle
n_{s}=\sqrt{\alpha_{1}\alpha_{2}k_{||}^{2}V_{rel}^{2}-\frac{B(k)^{2}}{2\pi(\rho_{1}+\rho_{2})}k_{||}^{2}-gk(\alpha_{1}-\alpha_{2})},$
(1)

where $\alpha_{1}=\frac{\rho_{1}}{\rho_{1}+\rho_{2}}$ and
$\alpha_{2}=\frac{\rho_{2}}{\rho_{1}+\rho_{2}}$ with $\rho_{1}$ and $\rho_{2}$
the mass density of the disk and the accreted gas, $g$ the vertical
gravitational acceleration, $V_{rel}$ is the relative velocity between the
accreted gas and the galactic disk gas, $B(k)$ the absolute value of the
magnetic field at a spatial scale $2\pi/k$, $k$ the absolute value of the
wavenumber, and $k_{||}$ the component of the wavenumber parallel to the
relative velocity.

The condition for instability in the presence of rotation was obtained by
Huppert (1968):

$k_{||}^{2}V_{rel}^{2}\geq 4\Omega^{2},$ (2)

where $\Omega$ is the angular velocity. As we shall see, this condition is
satisfied with a wide margin, and therefore the growth rate given by equation
(1) is appropriate here.

##  3 The astrophysical setting considered

We consider the case of a disk galaxy similar to the MW. We adopt parameters
consistent with the observations of Ho et al. (2017), Ho, Martin, & Turner
(2019), Martin et al. (2019), Ho & Martin (2020), Zabl et al. (2019), and the
simulations of Trapp et al. (2022). We consider a cold CGM gas that at a
galactic radius $R=40$ kpc has approximately the same rotational velocity as
that of the disk. We thus adopt a relative velocity which is radial (inward);
$V_{rel}=40$ km/ s. At the galaxy outskirt we take $\rho_{1}=1.67\times
10^{-25}$ g cm-3 corresponding to a number density of $0.1$ cm-3. The
inflowing gas density at the above radius was taken to be
$\rho_{2}=0.055\rho_{1}=9.2\times 10^{-27}$ g cm-3. This yields
$\alpha_{1}=0.95,\ \alpha_{2}=0.05$.

These parameters imply a mass accretion rate for an angular extension of the
inflowing gas, $0<\beta<2\pi$, and a scale height $H=330$ pc

$\dot{M}=\beta 2HR\rho_{2}V_{rel}=0.13\beta\ {\rm M_{\odot}\ yr^{-1}}$ (3)

which is quite reasonable.

###  3.1 The vertical acceleration g

The vertical acceleration is the sum of the vertical components of the
galactic acceleration and the self gravity of the disk:

$\displaystyle g=g_{gal}+g_{self},$ (4) $\displaystyle{\rm where}\
g_{gal}=\frac{v_{rotation}^{2}}{R}\frac{H}{R}\ \ ,\ {\rm and}\ g_{self}=2\pi
G\Sigma$

where $\Sigma$ is the surface mass density of the disk. At $R=40$ kpc we adopt
the parameters from Sofue (2013): $v_{rotation}=158\ {\rm km/s,\ \ {\rm and}\
\Sigma=1\ M_{\odot}\ pc^{-2}}$. These imply $g_{gal}=1.8\times 10^{-11}{\rm
cm\ s^{-2}}$ and $g_{self}=8.8\times 10^{-11}{\rm cm\ s^{-2}}$, yielding
$g=1.06\times 10^{-10}{\rm cm\ s^{-2}}$.

###  3.2 Random galactic magnetic field

Many observations were made with the aim of revealing the nature of the
galactic random magnetic field. Rand & Kulkarni (1989), Ohno & Shibata (1993),
Han, Ferriere, & Manchester (2004), and Han (2017) concluded that measurements
of the rotation and dispersion measures of pulsars can be well represented by
a random field with coherence length ("cell") $L_{B}$ and field strength
$B_{0}$. As a result, the average field along a given line of sight of length
$L=2\pi/k$ due to the randomly oriented in-cell fields is

$B(k)=B_{0}\sqrt{\frac{k}{k_{B}}},\ \ k\leq k_{B},\ \ {\rm where}\
k_{B}=\frac{2\pi}{L_{B}}.$ (5)

Substituting this to equation (1) yields

$\displaystyle
n_{s}=\sqrt{\alpha_{1}\alpha_{2}k_{||}^{2}V_{rel}^{2}-\frac{B_{0}^{2}}{2\pi(\rho_{1}+\rho_{2})}\frac{k_{||}^{3}}{k_{B}}-gk(\alpha_{1}-\alpha_{2})}.$
(6)

We adopt $B_{0}=1\ \mu$ G and $k_{B}=2\pi/(200\ {\rm pc})$. Substitution of
the numerical values shows that $n_{s}(k)$ is real and positive for
wavenumbers $1.31\times 10^{-22}{\rm cm^{-1}}<k<8.51\times 10^{-21}{\rm
cm^{-1}}$ corresponding to spatial scales $\ 0.24\ {\rm kpc}<2\pi/k<15.5\ {\rm
kpc}$.

##  4 turbulence model

Extending the earlier turbulence models of Canuto & Goldman (1985) and Canuto,
Goldman, & Chasnov (1987), we employ the model of Canuto, Goldman, &
Mazzitelli (1996) to derive the expected power spectrum of the turbulence. The
model is formulated as an integral equation which represents a balance between
the net rate of energy input to the turbulence in the wavenumber range
$(k_{0}-k)$ and the rate of the energy cascaded to all scales smaller than
$k$,

$\displaystyle\int_{k_{0}}^{k}n_{s}(k^{\prime})dk^{\prime}=y(k)\nu_{t}(k),$
(7) $\displaystyle{\rm where}\ \ \ \
y(k)=\int_{k_{0}}^{k}F(k^{\prime})k^{\prime 2}dk^{\prime}.$ (8)

$F(k)$ is the power spectrum of the turbulent velocity, $y(k)$ is the
$k$-space mean square vorticity at wavenumber $k$, $n_{s}(k)$ is the net rate
controlling the energy input from the source at $k$ incorporating the rate of
energy dissipation by molecular viscosity, and $\nu_{t}(k)$ is the turbulent
kinematic viscosity at wavenumber $k$ exerted by all the eddies with wave
number larger than $k$. Here, $k_{0}$ is the wavenumber corresponding to the
largest scale of the turbulence. The turbulent viscosity at wavenumber $k$ is
defined by

$\nu_{t}(k)=\int_{k}^{\infty}\frac{F(k^{\prime})}{n_{c}^{*}(k^{\prime})}dk^{\prime}.$
(9)

Here, $n_{c}^{*}(k)$ is the rate controlling the eddy nonlinear correlation
(heuristically the rate of the eddy breakup at $k$).

By differentiating equation (7) it is possible to obtain a rate equation:

$n_{s}(k)+\frac{y(k)}{n_{c}^{*}(k)}=\nu_{t}(k)k^{2}=\gamma n_{c}(k).$ (10)

The second term on the left hand side is the rate controlling the energy
cascaded from all spatial scales larger $2\pi/k$. The right hand side is the
rate at which energy is transferred to the smaller scales, with $\gamma$ a
positive dimensionless constant which was fitted by Canuto, Goldman, &
Mazzitelli (1996) to produce a Kolmogorov inertial power spectrum, yielding
$\gamma=0.088\left(K_{c}/1.5\right)^{-3}$. Here $K_{c}$ denotes the so-called
Kolmogorov constant.

The eddy correlation rate is modeled as

$\gamma n_{c}^{*}(k)=\left([\gamma
n_{c}(k)]^{1/2}+[n_{s}(k)]^{1/2}\right)^{2},$ (11)

and is dependent on both the turbulent viscosity and on $n_{s}(k)$. For a
given $n_{s}(k)$, the solution of the coupled equations yields the power
spectrum $F(k)$. The largest spatial scale corresponding to the smallest
wavenumber $k_{0}$ is obtained from

$\frac{d}{dk}\left(\frac{n_{s}(k)}{k^{2}}\right)_{k_{0}}=0,$ (12)

and at this scale $F(k_{0})=0$.

By combining equation (10) and (11), one obtains an algebraic expression of
$y(k)$ in term of $n_{s}(k)$ and $n_{c}(k)$:

$y(k)=\gamma^{-1}\left(\gamma n_{c}(k)-n_{s}(k)\right)\left([\gamma
n_{c}(k)]^{1/2}+[n_{s}(k)]^{1/2}\right)^{2}.$ (13)

Differentiating equation (9) gives

$\displaystyle F(k)=-\left([\gamma
n_{c}(k)]^{1/2}+[n_{s}(k)]^{1/2}\right)^{2}\frac{d}{dk}\left(\frac{n_{c}(k)}{k^{2}}\right)$
(14)

which upon using the relation $dy(k)/dk=F(k)k^{2}$ leads to

$\displaystyle\frac{dy(k)}{dk}=-k^{2}\left([\gamma
n_{c}(k)]^{1/2}+[n_{s}(k)]^{1/2}\right)^{2}\frac{d}{dk}\left(\frac{n_{c}(k)}{k^{2}}\right),$
(15)

From equations ( 13) and (15) one obtains a first-order differential equation
for $n_{c}(k)$. The initial condition is $\gamma n_{c}(k_{0})=n_{s}(k_{0})$.
Once $n_{c}(k)$ has been solved, equation (14) determines $F(k)$.

##  5 Computed power spectrum and the turbulent velocity

We take $n_{s}(k)$ to be the linear growth rate given by equation (6). As has
been argued by Canuto, Goldman, & Mazzitelli (1996), the self-consistent
formulation of the eddy correlation timescale effectively modifies the growth
rate and makes it dependent also on the turbulence. We obtain the velocity
power spectrum and the value of the turbulent velocity for two sets of
parameters that are within the range suggested by the observations and
simulations. It is of interest to identify which features are independent of
the precise values of the parameters.

###  5.1 Parameters set 1

For the parameters listed in the previous sections, equation (12) yields

$\displaystyle k_{0}=1.96\times 10^{-22}\ {\rm cm^{-1}\ and}$ (16)
$\displaystyle L_{0}=2\pi/k_{0}=10.4\ {\rm\ kpc}.$

The smallest scale $2\pi/k_{f}$ for which $n_{s}(k_{f})=0$ is $240$ pc. Note
that $L_{0}$, the largest scale of the turbulence, is about $2/3$ of the
largest scale at which $n_{s}>0$. The formal solution of the power spectrum
for $k<k_{0}$ is negative and thus no turbulence exists in this range even
though $n_{s}$ is positive there. The growth rate at $k=k_{0}$ is

$n_{s}(k_{0})=9.9\times 10^{-17}s^{-1}=\frac{1}{3.3\times 10^{8}\ yr}.$ (17)

The growth rate $n_{s}(k)$ and the power spectrum $F(k)$ are shown in Fig. 1
and Fig. 2, respectively. The power spectrum rises from $F(k_{0})=0$, reaches
a maximum at $k\simeq 3k_{0}$ corresponding to a spatial scale of $\simeq 3.3$
kpc and then declines with a logarithmic slope that changes from -1 to -1.2 as
$k$ increases.

The turbulent velocity is

$v_{turb}=\sqrt{\int_{k_{0}}^{k_{f}}F(k)dk}=32.6\gamma^{-1/2}\ {\rm km/s}.$
(18)

###  5.2 Parameters set 2

Here the computation is repeated for $V_{rel}=80$ km/s, which is at the high
end of the observational and simulated values. In order not to obtain an
unrealistically high mass accretion rate, a lower value of $\rho_{2}$ is
adopted: $\rho_{2}=3.06\times 10^{-27}\ {\rm g\ cm^{-3}}$. Thus,
$\alpha_{1}=0.982$ and $\alpha_{2}=0.018$. The mass accretion rate for these
parameters is is

$\dot{M}=\beta 2HR\rho_{2}V_{rel}=0.1\beta M_{\odot}\ {\rm yr^{-1}},$ (19)

similar to that obtained for the previous parameter set. The largest
turbulence scale is larger (reflecting the larger value of $V_{rel}$):

$\displaystyle k_{02}=1.4\times 10^{-22}\ {\rm cm^{-1}}$ (20) $\displaystyle
L_{02}=2\pi/k_{0}=14.5\ {\rm kpc}.$

The smallest scale is $165$ pc. Since the coherence length of the magnetic
field was taken as $200$ pc, the power spectrum smallest scale was taken as
$200$ pc corresponding to a relative wave number $k/k_{0}=72.5$. The turbulent
velocity for these parameters is

$v_{turb}=\sqrt{\int_{k_{0}}^{k_{f}}F(k)dk}=46.5\gamma^{-1/2}\ {\rm km/s}.$
(21)

The growth rate $n_{s}(k)$ and the power spectrum $F(k)$ are shown in Fig. 3
and Fig. 4, respectively. We note that the power spectrum has the same
distinctive shape as in set 1 despite the different values of the spatial
scales and of the turbulent velocity.

##  6 discussion and conclusions

The power spectrum of the turbulence (Fig. 2 and Fig. 4) has a unique shape.
It spans wavenumbers corresponding to spatial scales in the range
($(10.4-0.22)\ {\rm kpc}$ in case 1. and $(14.5-0.20)\ {\rm kpc}$ for the
parameters of case 2. It is zero for the largest scale, rises and then
declines, almost with a constant logarithmic slope. For intermediate spatial
scales the power spectrum has a logarithmic slope of $-1.1$, which turns
gradually to a value of -2 for smaller scales.

While the regions where the power spectrum has a power-law dependence on the
wavenumber resembles an inertial range, this not the case, since $n_{s}(k)$ is
not zero in this range. Actually, the latter fact is the reason for the
shallow decline of the power spectrum (compared to the Kolmogorov spectral
logarithmic slope of $-5/3$). For $k$-values larger than the maximal $k$
(where the energy input vanishes) there may exist an inertial range
proportional to $k^{-5/3}$ provided that the microscopic viscosity can be
neglected.

The turbulent velocity is $32.6\ \gamma^{-1/2}{\rm km/s}$ for case 1. and
$46.5\ \gamma^{-1/2}{\rm km/s}$ for case 2. For the value of $\gamma\simeq
0.9$ obtained by Canuto, Goldman, & Mazzitelli (1996) the turbulent velocities
are quiet large. This is due to the wide range and slow decline of the power
spectrum.

We note that the time scale $n_{s}(k)^{-1}$ is shorter than the rotation
period at $R=40\ $ kpc. In addition, the timescales that characterize the
turbulence: $n_{c}(k)^{-1}$ and $n_{c}^{*}(k)^{-1}$ are much shorter.

Obviously, the numerical values depend on the assumed radial velocity and on
the value of $\rho_{2}$. The range of the spatial scales of the turbulence
changes as well as the normalization of the power spectrum and the turbulent
velocity. However, the unique shape of the power spectrum is unchanged, a
feature that identifies the turbulence as KH.

Observation of a fluctuating velocity field at the outskirts of a galaxy
having a large-scale power spectrum with the shape obtained here could serve
as indirect observational evidence for gas accretion from the CGM.

Figure 1: Set 1:The dimensionless growth rate $\eta_{s}\equiv
n_{s}(k)/n_{s}(k_{0})$ as a function of the dimensionless wavenumber $q\equiv
k/k_{0}$.

Figure 2: Set 1: The dimensionless power spectrum
$f\equiv F(k)\left(n_{s}(k_{0})/k_{0}\right)^{-2}\gamma$ as a function of
dimensionless wavenumber $q\equiv k/k_{0}$. The Orange line has a logarithmic
slope of $-1.2$; the Green line has a logarithmic slope of $-1$.

Figure 3: Set 2: the dimensionless growth rate $\eta_{s}\equiv
n_{s}(k)/n_{s}(k_{0})$ as a function of the dimensionless wavenumber $q\equiv
k/k_{0}$.

Figure 4: Set 2: The dimensionless power spectrum
$f\equiv F(k)~{}\left(n_{s}(k_{0})/k_{0}\right)^{-2}\gamma$ as a function of
dimensionless wavenumber $q\equiv k/k_{0}$. The Orange line has a logarithmic
slope of $-1.2$; the Green line has a logarithmic slope of $-1$.

## Acknowledgements

Itzhak Goldman thanks Afeka College, as well as the Astrophysics department of
Tel Aviv University.

##  7 data availability

This is a theoretical paper. No specific data is analyzed.

## References

  * Bland-Hawthorn et al. (2017) Bland-Hawthorn J., Maloney P. R., Stephens A., Zovaro A., Popping A., 2017, ApJ, 849, 51. doi:10.3847/1538-4357/aa8f45
  * Canuto & Goldman (1985) Canuto V. M., Goldman I., 1985, PhRvL, 54, 430. doi:10.1103/PhysRevLett.54.430
  * Canuto, Goldman, & Chasnov (1987) Canuto V. M., Goldman I., Chasnov J., 1987, PhFl, 30, 3391. doi:10.1063/1.866472
  * Canuto, Goldman, & Mazzitelli (1996) Canuto V. M., Goldman I., Mazzitelli I., 1996, ApJ, 473, 550. doi:10.1086/178166
  * Chandrasekhar (1961) Chandrasekhar S., 1961, hhs..book
  * Das et al. (2020) Das S., Sardone A., Leroy A. K., Mathur S., Gallagher M., Pingel N. M., Pisano D. J., et al., 2020, ApJ, 898, 15. doi:10.3847/1538-4357/ab97b9
  * Elmegreen, Elmegreen, & Leitner (2003) Elmegreen B. G., Elmegreen D. M., Leitner S. N., 2003, ApJ, 590, 271. doi:10.1086/374860
  * Elmegreen, Struck, & Hunter (2014) Elmegreen B. G., Struck C., Hunter D. A., 2014, ApJ, 796, 110. doi:10.1088/0004-637X/796/2/110
  * Elmegreen et al. (2016) Elmegreen D. M., Elmegreen B. G., Sánchez Almeida J., Muñoz-Tuñón C., Mendez-Abreu J., Gallagher J. S., Rafelski M., et al., 2016, ApJ, 825, 145. doi:10.3847/0004-637X/825/2/145
  * Elmegreen (2016) Elmegreen B. G., 2016, IAUS, 317, 204.
doi:10.1017/ S1743921315006948

  * Fleck (1983) Fleck R. C., 1983, ApJ, 270, 507. doi:10.1086/161143
  * Fleck (1984) Fleck R. C., 1984, AJ, 89, 506. doi:10.1086/113542
  * Fleck (1989) Fleck R. C., 1989, AJ, 97, 783. doi:10.1086/115023
  * Fleck (2020) Fleck R., 2020, Natur, 583, E24. doi:10.1038/s41586-020-2476-5
  * Gómez & Ostriker (2005) Gómez G. C., Ostriker E. C., 2005, ApJ, 630, 1093. doi:10.1086/432086
  * Han, Ferriere, & Manchester (2004) Han J. L., Ferriere K., Manchester R. N., 2004, ApJ, 610, 820. doi:10.1086/421760
  * Han (2017) Han J. L., 2017, ARA&A, 55, 111. doi:10.1146/annurev-astro-091916-055221
  * Ho & Martin (2020) Ho S. H., Martin C. L., 2020, ApJ, 888, 14. doi:10.3847/1538-4357/ab58cd
  * Ho, Martin, & Turner (2019) Ho S. H., Martin C. L., Turner M. L., 2019, ApJ, 875, 54. doi:10.3847/1538-4357/ab0ec2
  * Ho et al. (2017) Ho S. H., Martin C. L., Kacprzak G. G., Churchill C. W., 2017, ApJ, 835, 267. doi:10.3847/1538-4357/835/2/267
  * Huppert (1968) Huppert H. E., 1968, JFM, 33, 353. doi:10.1017/S0022112068001357
  * Ianjamasimanana et al. (2022) Ianjamasimanana R., Koribalski B. S., Józsa G. I. G., Kamphuis P., de Blok W. J. G., Kleiner D., Namumba B., et al., 2022, MNRAS, 513, 2019. doi:10.1093/mnras/stac936
  * Kacprzak (2017) Kacprzak G. G., 2017, ASSL, 430, 145. doi:10.1007/978-3-319-52512-9_7
  * Kereš et al. (2005) Kereš D., Katz N., Weinberg D. H., Davé R., 2005, MNRAS, 363, 2. doi:10.1111/j.1365-2966.2005.09451.x
  * Lockman (2017) Lockman F. J., 2017, ASSL, 430, 49. doi:10.1007/978-3-319-52512-9_3
  * Luo et al. (2021) Luo Y., Heckman T., Hwang H.-C., Rowlands K., Sánchez-Menguiano L., Riffel R., Bizyaev D., et al., 2021, ApJ, 908, 183. doi:10.3847/1538-4357/abd1df
  * Mandelker et al. (2016) Mandelker N., Padnos D., Dekel A., Birnboim Y., Burkert A., Krumholz M. R., Steinberg E., 2016, MNRAS, 463, 3921. doi:10.1093/mnras/stw2267
  * Martin et al. (2019) Martin C. L., Ho S. H., Kacprzak G. G., Churchill C. W., 2019, ApJ, 878, 84. doi:10.3847/1538-4357/ab18ac
  * Ohno & Shibata (1993) Ohno H., Shibata S., 1993, MNRAS, 262, 953. doi:10.1093/mnras/262.4.953
  * Rand & Kulkarni (1989) Rand R. J., Kulkarni S. R., 1989, ApJ, 343, 760. doi:10.1086/167747
  * Richter (2017) Richter P., 2017, ASSL, 430, 15. doi:10.1007/978-3-319-52512-9_2
  * Sofue (2013) Sofue Y., 2013, PASJ, 65, 118. doi:10.1093/pasj/65.6.118
  * Trapp et al. (2022) Trapp C. W., Kereš D., Chan T. K., Escala I., Hummels C., Hopkins P. F., Faucher-Giguère C.-A., et al., 2022, MNRAS, 509, 4149. doi:10.1093/mnras/stab3251
  * Tumlinson, Peeples, & Werk (2017) Tumlinson J., Peeples M. S., Werk J. K., 2017, ARA&A, 55, 389. doi:10.1146/annurev-astro-091916-055240
  * Vulcani et al. (2018) Vulcani B., Poggianti B. M., Moretti A., Mapelli M., Fasano G., Fritz J., Jaffé Y., et al., 2018, ApJ, 852, 94. doi:10.3847/1538-4357/aa992c
  * Zabl et al. (2019) Zabl J., Bouché N. F., Schroetter I., Wendt M., Finley H., Schaye J., Conseil S., et al., 2019, MNRAS, 485, 1961. doi:10.1093/mnras/stz392


# ViSTRA2: Video Coding using Spatial Resolution and Effective Bit Depth
Adaptation

Fan Zhang, , Mariana Afonso, , and David R. Bull Manuscript drafted at
September 2019.Fan Zhang, Mariana Afonso, and David R. Bull are with the
Visual Information Laboratory, University of Bristol, Bristol, UK.E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
###### Abstract

We present a new video compression framework (ViSTRA2) which exploits
adaptation of spatial resolution and effective bit depth, down-sampling these
parameters at the encoder based on perceptual criteria, and up-sampling at the
decoder using a deep convolution neural network. ViSTRA2 has been integrated
with the reference software of both the HEVC (HM 16.20) and VVC (VTM 4.01),
and evaluated under the Joint Video Exploration Team Common Test Conditions
using the Random Access configuration. Our results show consistent and
significant compression gains against HM and VVC based on Bjønegaard Delta
measurements, with average BD-rate savings of 12.6% (PSNR) and 19.5% (VMAF)
over HM and 5.5% (PSNR) and 8.6% (VMAF) over VTM.

###### Index Terms:

Spatial resolution adaptation, effective bit depth adaptation, video
compression, machine learning based compression, HEVC, VVC

##  I Introduction

The development of video technology for film, television, terrestrial and
satellite transmission, surveillance, and especially Internet video, demands
much higher bandwidth than ever before to support new services and ever
increasing numbers of users consuming ever more content. Although
communication techniques have improved significantly recently due to the
advances in network and physical layers, the bitrate available to an
individual user at the application layer is still limited, due to the
increasing amounts of video data (with higher quality and more immersive
formats) consumed everyday. Video compression offers the solution to this
problem, but equally presents major challenges in how we achieved yet higher
coding gains.

Since the first video coding standard, H.120 [1], was introduced in 1984,
standardisation of video formats and coding algorithms have played an
important role in the application and success of video technology [2]. In the
last three decades, video compression standards have been improved
significantly, with each new standard providing approximately twice the
compression efficiency compared to its predecessor. The latest effort,
Versatile Video Coding (VVC) [3], initiated in 2018, is targeting 30-50%
overall bitrate savings over the current standard, High Efficiency Video
Coding (HEVC), through integrating more sophisticated tools.

Inspired by recent advances in artificial intelligence, machine learning-based
methods have seen increasing utility in video compression algorithms both for
end to end compression and to enhance conventional coding tools [4, 5, 6].
Although some of these works, particularly those using Convolutional Neural
Networks (CNNs), have reported evident coding gains, very few of them have
been adopted by standards due to their high complexity and unconventional
architectures needed for the CNN models [7].

Alongside conventional coding tools, spatial and temporal resolution
adaptation have also been employed in video compression to improve coding
efficiency. This type of approach has previously been adopted for relatively
low bitrate scenarios [8, 9], as the quality of up-sampled video content can
be inconsistent at higher bitrates due to the blurring or aliasing artefacts
introduced by resolution adaptation. It is also noted that, in these
approaches, only spatial and temporal (frame rate) resolution adaptations have
been exploited for video compression and there is little reported work on bit
depth adaptation.

Based on our previous work on spatial resolution (SR) and effective bit depth
(EBD) [10, 11, 12, 13, 14, 15] adaptations, and perceptual quality assessment
[16], a new video compression framework, ViSTRA2, is proposed in this paper.
It dynamically down-samples the spatial resolution and bit depth of the input
video at the encoder, and reconstructs its original resolution and bit depth
during decoding. To improve the final reconstruction quality, a deep CNN is
employed for both SR and EBD up-sampling, trained on a large, multi-resolution
video database for HEVC and VVC compression at various quantisation levels.
The proposed approach has been integrated with the reference test models of
HEVC (HM 16.20) and VVC (VTM 4.01), and evaluated on the Joint Video
Exploration Team (JVET) Common Test Conditions (CTC). The results exhibit
significant coding gains over both HEVC (HM) and VVC (VTM), with average BD-
rate gains (based on PSNR) of 12.6% and 5.5% respectively.

The main contributions of this paper are summarised as follows:

  1. 1.

Integration of both spatial resolution (SR) and effective bit depth (EBD)
adaptation into one coding framework.

  2. 2.

The employment of a single deep CNN for both SR and EBD up-sampling (using
different model parameters for various scenarios);

  3. 3.

Robust spatial resolution adaptation decisions based on a bespoke SR-dependent
quality metric.

  4. 4.

Demonstration that using deep networks in an end to end system offers
flexibility in the distribution of complexity across that system.

Comparing to this paper, our previous contributions [11, 12, 13, 14] focus
solely on the adaptation of spatial resolution or effective bit depth. In
addition, the employed CNN architecture in this paper has been enhanced
compared to that in [10, 11, 12]. Furthermore, an extended training database
has been employed here to achieve improved reconstruction results.
Comprehensive evaluation results are presented for ViSTRA2, against both HEVC
and VVC using various quality assessment methods. An early version of ViSTRA2
was contributed by the University of Bristol (JVET-J0031) to the JVET “Call
for Proposals” [17] for Versatile Video Coding (VVC).

The remainder of this paper is organised as follows. Section II reviews the
state-of-the-art of video coding algorithms, specifically covering the
approaches based on resolution adaptation and deep learning. In Section II,
the proposed coding framework is described in detail alongside its important
components. The performance of the presented work is fully evaluated in
Section IV, while Section V provides the conclusions and outlines future work.

##  II Background

This section is divided into three subsections. The first overviews the
development of video compression standards and the recent advances on royalty-
free coding formats. Research work on machine learning based video coding is
then briefly reviewed, followed by a summary of compression algorithms
employing resolution adaptation.

###  II-A Video Coding Standards: A Brief Overview

Since the early 1980s, video compression has been subject to global
standardisation. Each standard defines the bitstream format and the decoder
structure, alongside the associated test model (reference encoder), which
generates standard-compliant streams and provides benchmark coding
performance.

One of the most successful examples, H.264/AVC (Advanced Video Coding) [18]
was jointly delivered in 2004 by ITU-T (VCEG) and ISO/IEC (MPEG). It remains
the most widely adopted coding standard for Internet streaming, HDTV and Blu-
ray players, although its successor H.265/HEVC [19], developed in 2013, offers
nearly double the coding efficiency. More recently, aiming to achieve even
higher compression performance (30%-50%) than HEVC, a new coding standard,
Versatile Video Coding (VVC) [3] is under development, offering better support
for immersive video including high dynamic range and 360°.

In order to provide open source and royalty-free solutions for media delivery,
the Alliance for Open Media (AOM) [20] was formed in 2015 by a consortium of
companies. Its first generation codec, AV1 (AOMedia Video 1) [21], was
released in 2018, and is considered one of the primary competitors of MPEG
standards. Other notable coding standards also include Essential Video
Coding/MPEG-5 [22], Microsoft WMV (Windows Media Video) [23], BBC (British
Broadcasting Corporation) Dirac [24], and AVS standards [25].

###  II-B Machine Learning based Video Compression

With recent advances in computational equipment and graphics processing (GPU)
devices, deep neural networks, especially convolutional neural networks
(CNNs), now offer tractable solutions to many image processing problems. They
are being increasingly applied in image and video compression to enhance
various coding tools including intra prediction [26, 27, 28, 29], motion
estimation [30, 31, 32, 33], transforms [34], quantisation [35], entropy
coding [36, 37] and loop filtering [38, 39, 40, 41]. Detailed reviews on
machine learning based compression can be found in [4, 5, 6]. Among the
responses to the Call for Proposals for VVC [42], there were five proposals
containing coding tools based on neural networks, but few of these have been
adopted by VVC [3], though its development is still ongoing. This is mainly
due to their relatively high computational complexity, especially when CNN-
based calculation has to be executed at the decoder.

###  II-C Resolution Adaptation for Compression

In video compression, resolution adaptation based approaches are utilised to
trade off the relationship between quantisation and (spatial and temporal)
resolutions. These methods encode a lower resolution version of a video, and
reconstruct its original resolution during decoding. This process can be
applied at different coding stages – for each macroblock or Coding Tree Unit
(CTU) [43, 8, 44], frame [45, 11], group of pictures [46, 10], or sequence [9,
47]. The reconstruction quality is highly content dependent and relies on the
up-sampling methods employed. This is why this type of approach was mainly
applicable in low bitrate cases [45, 9, 8] or for intra coding [11], when
simple interpolation filters were employed. Inspired by the recent development
of (especially CNN-based) super-resolution algorithms [48, 49, 50, 51], it is
now possible to extend these approaches to higher bitrate ranges with improved
reconstruction quality.

Due to the content-dependent nature of these adaptation approaches, it is also
important to characterise the relationship between spatial resolution,
quantisation and visual quality. For this purpose, subjective video databases
[52, 53], were developed to investigate perceptual quality across a range of
spatial resolutions, quantisation levels and up-sampling methods. Based on
these databases, objective quality metrics have been designed for generic [53,
54] or bespoke [16] applications, which can facilitate quantisation resolution
optimisation during resolution adaptation inside the coding loop.

##  III Proposed Algorithm

Figure 1: Diagram of the ViSTRA2 encoder.

The proposed ViSTRA2 coding framework is illustrated in Fig. 1. At the
encoder, the spatial resolution (SR) is firstly determined by a Quantisation-
Resolution Optimisation (QRO) module according to the input content and the
initial base quantisation parameter ($\mathrm{QP}_{\mathrm{base}}$) values
configured in the host encoder. The original video frames are then spatially
down-sampled (by 2 in this case111Greater improvement may be achieved by
applying multiple re-sampling ratios as in [12]. This will however requires
more sophisticated CNN training process and accurate QRO prediction. In this
paper, as effective bit depth adaptation has also been employed in the coding
workflow, a single spatial resolution re-sampling ratio is used for algorithm
simplification.) if SR adaptation is enabled, by applying an anti-aliasing
modified separable Lanczos3 filter [55] with a kernel width of 12. In order to
signal the SR adaptation decision to the decoder, a flag bit is inserted into
the bitsteam as side information.

Secondly, the effective bit depth (EBD) (for both luma and chroma channels) is
down-sampled by 1 bit prior to encoding through bitwise right shift. Here EBD
is defined as the actual bit depth used to represent the video content, which
is different from the coding bit depth (CBD) defined as the pixel bit depth,
e.g. InternalBitDepth in HEVC HM and VVC VTM reference encoders. Throughout
the coding process, EBD is lower than or equal to CBD in the proposed coding
workflow, while CBD remains constant. It is noted that there is no
optimisation module for EBD adaptation here. This is because, through
observing the coding results on the training data and the preliminary results
generated in [14], in most cases, CNN-based EBD adaptation can provide
improved (or equivalent) compression performance. Therefore, EBD adaptation is
always enabled in the coding framework of ViSTRA2, and does not require any
side information indicating the decoder the EBD changes.

During encoding, to obtain similar bitrate ranges and facilitate comparison
with anchor codecs (with the same $\mathrm{QP}_{\mathrm{base}}$ values), a
fixed quantisation parameter (QP) offset is applied on
$\mathrm{QP}_{\mathrm{base}}$. This offset equals -6 when only EBD adaptation
is enabled, and becomes -12 if both SR and EBD adaptations are applied. These
two values were empirically obtained through the observation on the coding
statistics of training sequences [11, 14].

At the decoder, based on the value of the flag bit, decoded SR and EBD down-
sampled video frames are up-sampled to their original SR and EBD using a deep
CNN. If SR adaptation is enabled, the spatial resolution of the decoded video
frames is firstly up-sampled by 2 using a nearest neighbour filter before CNN
reconstruction222It is noted that, in our previous work [10, 12, 17], pre-CNN
up-sampling was conducted using a Lanczos3 filter. However we have found that
by using a nearest neighhour filter here the overall reconstruction
performance can be slightly improved.. The network architecture and training
process alongside the detail description of the QRO module are presented
below.

###  III-A The Employed CNN Architecture

Figure 2: CNN architecture employed for EBD (and SR) up-sampling.

The architecture of the convolutional neural network employed to reconstruct
full spatial resolution and effective bit depth is shown in Fig. 2. The input
of the CNN is a 96$\times$96 compressed RGB colour image block with reduced
EBD (and re-sampled SR if SR adaptation is enabled), while the output is
expected to be the corresponding original image block with the same size. This
architecture is modified based on the generator (SRResNet) of SRGAN [48],
which was developed for uncompressed image super-resolution. It starts from an
initial convolutional layer with a Parametric Rectified Linear Unit (PReLU) as
the activation function, and ends with another single convolutional layer with
a Tanh activation. Before the final output, a skip connection is employed
between the input of the CNN and the output the last convolutional layer.
Between these two convolutional layers, there are $N$ identical residual
blocks, one of each consisting of two convolutional layers and one PReLU in
between. In each residual block, a skip connection is applied between the
input of first convolutional layer and the output of the second. Another skip
connection is also employed between the output of the first convolutional
layer and the output of all residual blocks. In all convolutional layers, the
stride value is 1 and the kernel size equals 3$\times$3\. The number of
feature maps is 64 for most convolutional layers except the last one (3
feature maps there).

Comparing to the original architecture of SRResNet [48], two modifications
were applied:

  * •

Batch normalisation (BN) layers have not been used here, as they were found to
decrease image feature variability and influence overall performance [51].

  * •

The loss function employed for training the network is $\ell$1 loss rather
than $\ell$2 in [48]. This is based on recent work on CNN super-resolution
[56], which claims improved reconstruction quality achieved from this change.

###  III-B Network Training and Evaluation

The CNN employed for EBD and SR-EBD (if SR adaptation is enabled) up-sampling
was trained using 432 video clips at various spatial resolutions (108 source
sequences $\times$ 4 spatial resolutions), including 3840$\times$2160,
1920$\times$1080, 960$\times$540, 480$\times$270, each of them having 64
frames with a CBD (coding bit depth) of 10 bits. All clips were collected from
publicly available databases, including BVI-HFR [57], BVI-Texture [58],
Netflix Chimera [59] and Harmonic 4K [60]. Their EBD (by 1 bit) and SR-EBD
down-sampled (SR adaptation ratio of 2) versions were compressed using both
HEVC HM 16.20 and VVC VTM 4.01 encoders for five different initial base QP
values 22, 27, 32, 37 and 42 (QP offsets of -6 or -12 was applied during
encoding for EBD or SR-EBD version). The same coding configuration was used as
the Random Access (RA) mode (Main10 profile) in the Joint Video Exploration
Team (JVET) Common Test Conditions (CTC), with a fixed intra period of 64.

This results in 20 groups of reconstructed videos (2 codecs $\times$ 2
adaptation versions $\times$ 5 QP groups), each group containing 432
reconstructed sequences for a specific codec (HM or VTM), adaptation version
(EBD or SR-EBD) and QP group (22-42). For each SR-EBD down-sampled and
compressed video, its frames were spatially up-sampled to their original
spatial resolution using a nearest neighbour filter (see footnote 2). The
video frames of all reconstructed sequences in each group and their original
counterparts, were randomly selected and split into 96$\times$96 image blocks,
which were then converted to the RGB colour space. Block rotation has also
been employed for data augmentation to further enhance model generalisation.
For each training group, the total number of training image block pairs is
approximately 100,000, much more than that of our previous training sets in
[10, 14].

The training process was conducted in the Tensorflow environment [61], using
the following parameters: batch size of 16, learning rate of 0.0001, Adam
optimisation [62], weight decay of 0.1 and a total number of 200 training
epochs. The number of residual blocks ($N$) is fixed as 16, which is the same
as that used in [48]. This generates 20 different CNN models
($\mathrm{model}_{c,v,q}$), each for one group. Here $c$ stands for the used
codec (HM or VTM), $v$ represents adaptation versions (EBD or SR-EBD), and $q$
is denoted to the QP group. For a specific codec and adaptation version, the
CNN model used in evaluation depends on the initial (before applying the
offset) base QP ($\mathrm{QP_{base}}$):

$\left\\{\begin{array}[]{l r }\mathrm{model}_{c,v,22},\
\text{if}&\mathrm{QP_{base}}\leq 24.5\\\ \mathrm{model}_{c,v,27},\
\text{if}&24.5<\mathrm{QP_{base}}\leq 29.5\\\ \mathrm{model}_{c,v,32},\
\text{if}&29.5<\mathrm{QP_{base}}\leq 34.5\\\ \mathrm{model}_{c,v,37},\
\text{if}&34.5<\mathrm{QP_{base}}\leq 39.5\\\ \mathrm{model}_{c,v,42},\
\text{if}&\mathrm{QP_{base}}\geq 39.5\end{array}\right.$ (1)

When these CNN models were employed for evaluation, each EBD or SR-EBD down-
sampled frame (pre-CNN SR up-sampling has been applied here if SR adaptation
is enabled) is firstly segmented into 96$\times$96 overlapping blocks with an
overlap size of 4 pixels as CNN input (after converting to the RGB space). The
output full SR and EBD blocks were then aggregated back following the same
segmentation pattern to form the final reconstructed frame.

###  III-C Quantisation-Resolution Optimisation

Spatial resolution adaptation does not always lead to coding improvement,
being highly dependent on the original resolutions, spatio-temporal
characteristics and the host codec. Our previous work [10] employed four video
features to predict QP thresholds, beyond which encoding lower resolution
content produces higher compression efficiency. In ViSTRA2, an improved
machine learning based approach has been developed to make decisions on
resolution adaptation, based on a spatial resolution dependent quality metric,
SRQM [16], temporal information (TI) and initial base quantisation parameter.

SRQM is an efficient, full reference objective video quality metric, which
characterises the relationship between variations in spatial resolution and
visual quality. It employs wavelet decomposition, subband combination with
perceptually inspired weights, and spatio-temporal pooling to estimate the
relative quality between the frames of a high resolution reference video, and
one that has been spatially adapted through a combination of down- and up-
sampling. In this work, SRQM is applied between the uncompressed, original
resolution video frames and their re-sampled versions (using Lanczos3 filter
for both down- and up- sampling). The temporal information (TI) here is
defined as the average absolute frame difference between luma pixels in the
current frame and its neighbouring ones (one frame before and another after if
available).

Both SRQM and TI values were computed for all frames in the 432 uncompressed
sequences in the training dataset at various resolutions from 3840$\times$2160
to 480$\times$270\. This database does not include any test sequences used in
Section IV, and was used for training the CNN in Section III-B. The average
SRQM and TI values for each sequence, together with tested initial base QP
values (22, 27, 32, 37 and 42), were employed as input features to train a
fully connected, shadow neural network, which contains a hidden layer and an
output layer.

The target binary output of the network was generated through comparing the
rate-distortion performance (PSNR on luma values was used here as a quality
metric) of the original anchor codecs (HM and VTM) and that of ViSTRA2 (with
the trained CNN models) for all five tested initial base QPs on these training
sequences. For a training sequence and $\mathrm{QP_{base}}$, if the rate-PSNR
point of ViSTRA2 codec is above the rate-PSNR curve of the corresponding
anchor codec, the binary decision (for this sequence and QP) is defined as 1 –
the adaptation should be applied. Otherwise, it is configured to 0. This
results in 432$\times$5 binary decisions for each codec. The network was
trained offline using the Matlab function train, and two sets of network model
parameters were then obtained for HEVC HM 16.20 and VVC VTM 4.01.

When these models were employed in ViSTRA2 to predict whether spatial
resolution should be enabled, the average feature values used for the QRO are
obtained for a number of frames – one GOP for Random Access mode in HEVC HM
and VVC VTM – in order to maintain the same latency. The decision is applied
to the frames assessed. Due to the possibility of content variations (although
not very common in the JVET SDR test set), the QRO module may lead to frequent
resolution changes during encoding. This should not cause any problem if
resolution changing is supported at the frame or GOP level in the encoding
loop. It is noted that although an Ad-Hoc Group was established by JVET to
investigate resolution adaptivity [63], this feature had not been formally
adopted by VVC when this paper was written. Video sequence segmentation is
therefore executed in the current version of ViSTRA2, when different
resolutions have to be applied during encoding. If a video sequence has to be
segmented into more than one segments, their bitstreams are generated
sequentially by the host encoder. In order to avoid frequent sequence
segmentation, each video segment is constrained here to be longer than 1
second. A flag bit is inserted within the header of the bitstream for each
video segment to indicate the decoder resolution change. This solution will be
modified when resolution adaptivity feature is integrated into standardised
coding algorithms.

##  IV Results and Discussion

TABLE I: BD-rate results of ViSTRA2 when HM 16.20 was employed as host codec. Metric | PSNR | VMAF
---|---|---
Codec | ViSTRA2-w/o CNN | EBDA-CNN | ViSTRA2-CNN | ViSTRA2-w/o CNN | EBDA-CNN | ViSTRA2-CNN
Class-Sequence | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs
A-Campfire | -28.0% | +2.0% | -22.0% | -16.7% | -35.5% | -19.9% | -28.9% | -19.6% | -24.2% | -25.5% | -42.2% | -41.2%
A-FoodMarket4 | -12.2% | -3.5% | -7.9% | -7.2% | -18.0% | -11.8% | -14.4% | -12.5% | -12.2% | -12.2% | -24.5% | -20.9%
A-Tango2 | -11.9% | -6.8% | -8.6% | -9.7% | -17.4% | -15.2% | -11.9% | -10.0% | -12.5% | -15.4% | -22.3% | -20.7%
A-CatRobot1 | -4.4% | -0.3% | -11.9% | -12.3% | -15.1% | -13.1% | -6.3% | -3.7% | -18.4% | -22.2% | -23.1% | -23.7%
A-DaylightRoad2 | +1.1% | +2.4% | -10.6% | -14.3% | -11.2% | -13.7% | -0.5% | -2.1% | -17.5% | -25.3% | -21.7% | -26.5%
A-ParkRunning3 | -26.5% | -17.4% | -18.5% | -13.7% | -31.2% | -25.2% | -29.7% | -27.2% | -21.7% | -20.4% | -36.7% | -34.9%
Class A | -13.7% | -4.0% | -13.2% | -12.3% | -21.4% | -16.5% | -15.3% | -12.5% | -17.8% | -20.2% | -28.4% | -28.0%
B-BQTerrace | -0.3% | -0.2% | -11.0% | -9.8% | -11.0% | -9.8% | -1.1% | -2.8% | -21.7% | -28.2% | -21.7% | -28.2%
B-BasketballDrive | -1.3% | -1.7% | -11.0% | -10.5% | -10.0% | -10.5% | -2.9% | -4.9% | -11.7% | -14.2% | -12.6% | -14.2%
B-Cactus | -0.4% | -0.3% | -9.5% | -9.8% | -9.5% | -9.8% | -3.7% | -4.9% | -15.2% | -18.6% | -16.8% | -18.6%
B-MarketPlace | -0.7% | +2.3% | -4.8% | -4.5% | -6.7% | -4.7% | -3.3% | -0.8% | -12.4% | -13.7% | -18.0% | -15.9%
B-RitualDance | -1.8% | +1.2% | -8.0% | -7.0% | -9.8% | -7.2% | -2.0% | -1.2% | -12.7% | -13.6% | -16.1% | -14.7%
Class B | -0.9% | +0.3% | -8.9% | -8.3% | -9.4% | -8.4% | -2.6% | -2.9% | -14.7% | -17.7% | -17.0% | -18.3%
C-BQMall | +0.1% | +1.6% | -9.5% | -8.7% | -9.5% | -8.7% | -1.0% | -0.5% | -13.1% | -13.3% | -13.1% | -13.3%
C-BasketballDrill | -2.1% | +1.2% | -11.9% | -8.8% | -11.9% | -8.8% | -5.5% | -4.4% | -15.6% | -13.8% | -15.6% | -13.8%
C-PartyScene | +0.0% | +1.6% | -8.0% | -7.4% | -8.0% | -7.4% | -1.1% | -1.1% | -11.2% | -12.4% | -11.2% | -12.4%
C-RaceHorses | -3.5% | -2.0% | -10.5% | -8.4% | -10.5% | -8.4% | -4.6% | -5.4% | -12.7% | -13.4% | -12.7% | -13.4%
Class C | -1.4% | +0.6% | -10.0% | -8.3% | -10.0% | -8.3% | -3.0% | -2.9% | -13.1% | -13.2% | -13.1% | -13.2%
D-BQSquare | +0.8% | +2.1% | -15.6% | -15.8% | -15.6% | -15.8% | -1.3% | -2.3% | -18.1% | -22.7% | -18.1% | -22.7%
D-BasketballPass | -3.2% | -0.8% | -11.7% | -9.8% | -11.7% | -9.8% | -4.7% | -4.3% | -13.1% | -12.3% | -13.1% | -12.3%
D-BlowingBubbles | -0.4% | +1.3% | -8.0% | -7.1% | -8.0% | -7.1% | -2.4% | -1.3% | -12.5% | -12.0% | -12.5% | -12.0%
D-RaceHorses | -3.0% | -1.2% | -11.2% | -9.3% | -11.2% | -9.3% | -5.2% | -5.4% | -14.0% | -14.2% | -14.0% | -14.2%
Class D | -1.5% | +0.3% | -11.6% | -10.5% | -11.6% | -10.5% | -3.4% | -3.3% | -14.4% | -15.3% | -14.4% | -15.3%
Overall | -5.2% | -1.0% | -11.1% | -10.0% | -13.8% | -11.4% | -6.9% | -6.0% | -15.3% | -17.0% | -19.3% | -19.7%
| -3.1% | -10.6% | -12.6% | -6.5% | -16.2% | -19.5%
TABLE II: BD-rate results of ViSTRA2 when VTM 4.01 was employed as host codec. Metric | PSNR | VMAF
---|---|---
Codec | ViSTRA2-w/o CNN | EBDA-CNN | ViSTRA2-CNN | ViSTRA2-w/o CNN | EBDA-CNN | ViSTRA2-CNN
Class-Sequence | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs | H-QPs | L-QPs
A-Campfire | -20.2% | -9.0% | -11.4% | -11.3% | -30.3% | -16.8% | -24.6% | -21.3% | -16.7% | -20.9% | -38.3% | -36.5%
A-FoodMarket4 | -3.9% | +2.0% | -2.5% | -0.6% | -7.3% | -1.9% | -11.0% | -6.2% | -5.6% | -3.4% | -14.7% | -9.6%
A-Tango2 | -4.6% | +0.1% | -3.0% | -1.9% | -7.9% | -3.9% | -9.4% | -5.1% | -5.5% | -5.2% | -12.3% | -8.4%
A-CatRobot1 | +4.0% | +4.7% | -5.3% | -3.6% | -4.8% | -3.3% | +0.4% | -0.0% | -7.3% | -7.9% | -9.0% | -8.0%
A-DaylightRoad2 | +1.9% | +2.9% | -4.1% | -5.8% | -5.0% | -5.8% | -0.3% | -1.5% | -7.1% | -11.1% | -9.1% | -11.1%
A-ParkRunning3 | -19.1% | -17.0% | -15.7% | -17.0% | -24.0% | -25.0% | -22.1% | -19.7% | -16.7% | -16.3% | -26.6% | -25.7%
Class A | -7.0% | -2.7% | -7.0% | -6.7% | -13.2% | -9.5% | -11.2% | -9.0% | -9.8% | -10.8% | -18.3% | -16.5%
B-BQTerrace | +1.6% | +4.5% | -1.7% | +0.1% | -1.7% | +0.1% | +0.3% | +0.8% | -1.5% | -3.8% | -1.5% | -3.8%
B-BasketballDrive | -0.8% | +0.3% | -4.0% | -3.9% | -4.0% | -3.9% | -2.6% | -3.5% | -3.4% | -5.1% | -3.4% | -5.1%
B-Cactus | -1.5% | -0.4% | -4.4% | -4.0% | -4.4% | -4.0% | -3.7% | -4.6% | -7.3% | -8.7% | -7.3% | -8.7%
B-MarketPlace | +7.1% | +7.3% | +4.4% | +4.2% | +4.4% | +4.2% | +5.3% | +4.1% | -0.7% | -1.8% | -0.7% | -1.8%
B-RitualDance | +0.4% | +2.0% | -3.2% | -2.2% | -3.2% | -2.2% | -1.3% | -1.9% | -6.7% | -6.7% | -6.7% | -6.7%
Class B | +1.4% | +2.7% | -1.8% | -1.2% | -1.8% | -1.2% | -0.4% | -1.0% | -3.9% | -5.2% | -3.9% | -5.2%
C-BQMall | +3.6% | +4.9% | -2.2% | -2.1% | -2.2% | -2.1% | +2.3% | +2.1% | -2.5% | -2.8% | -2.5% | -2.8%
C-BasketballDrill | +3.3% | +7.2% | -1.8% | +0.7% | -1.8% | +0.7% | -0.7% | +0.3% | -3.3% | -2.7% | -3.3% | -2.7%
C-PartyScene | +0.9% | +2.6% | -4.3% | -3.2% | -4.3% | -3.2% | -0.3% | +0.4% | -4.8% | -3.7% | -4.8% | -3.7%
C-RaceHorses | -1.4% | -0.6% | -4.0% | -3.3% | -4.0% | -3.3% | -3.4% | -3.9% | -6.3% | -6.9% | -6.3% | -6.9%
Class C | +1.6% | +3.5% | -3.1% | -2.0% | -3.1% | -2.0% | -0.5% | -0.2% | -4.2% | -4.0% | -4.2% | -4.0%
D-BQSquare | +5.7% | +11.1% | -5.7% | -1.8% | -5.7% | -1.8% | +3.9% | +12.1% | -1.8% | +6.4% | -1.8% | +6.4%
D-BasketballPass | +0.1% | +1.9% | -5.6% | -4.6% | -5.6% | -4.6% | -0.6% | -0.6% | -6.2% | -5.4% | -6.2% | -5.4%
D-BlowingBubbles | +1.1% | +2.6% | -3.5% | -2.6% | -3.5% | -2.6% | -0.5% | +0.0% | -5.4% | -4.0% | -5.4% | -4.0%
D-RaceHorses | -1.9% | -0.5% | -6.5% | -5.6% | -6.5% | -5.6% | -5.9% | -6.0% | -10.3% | -10.4% | -10.3% | -10.4%
Class D | +1.3% | +3.8% | -5.3% | -3.7% | -5.3% | -3.7% | -0.8% | +1.4% | -5.9% | -3.4% | -5.9% | -3.4%
Overall | -1.2% | +1.4% | -4.5% | -3.6% | -6.4% | -4.5% | -3.9% | -2.9% | -6.3% | -6.3% | -9.0% | -8.2%
| 1.3% | -4.1% | -5.5% | -3.4% | -6.3% | -8.6%

The ViSTRA2 coding framework has been integrated into the HEVC (HM 16.20) and
VVC (VTM 4.01) reference software, and has been fully tested under JVET CTC
[64] using the Random Access configuration (Main10 profile). In order to
evaluate the proposed approach across a wider bitrate range, the initial base
QPs tested include 22, 27, 32, 37 and 42. The SDR (standard dynamic range)
video classes A1, A2, B, C and D from JVET CTC were employed here as test
content, none of which were utilised in the training of CNN and QRO modules in
Section III.

The rate quality performance of the integrated ViSTRA2 has been compared to
the corresponding reference codecs. Bjøntegaard delta measurements (BD) [65]
results were generated for both low QP (22, 27, 32 and 37) and high QP (27,
32, 37 and 42) values over all frames using two quality metrics (on luma
values only), Peak Signal-to-Noise Ratio (PSNR) and Video Multimethod
Assessment Fusion (VMAF, version 0.6.1) [54]. PSNR is the most commonly used
image quality metric for evaluating video compression performance, while the
recently developed VMAF is a machine learning based assessment method,
combining the Visual Information Fidelity measures (VIF) [66], Detail Loss
Metric (DLM) [67] and temporal frame difference [54] together using a Support
Vector Machine (SVM) regressor [68]. As the content generated by ViSTRA2
contains both compression distortions and resolution re-sampled artefacts,
VMAF, which is one of few quality metrics being trained and evaluated on
compressed and resolution adapted content, is expected to provide more
reliable assessment results (better correlation with subjective opinions) than
PSNR.

In order to further benchmark the contribution of CNN reconstruction, results
through up-sampling using Lanczos3 [55] (for SR) and simple bitwise left
shifting (for EBD) were also produced (denoted ViSTRA2-w/o CNN), based on the
same QRO decisions (as CNN-based ViSTRA2). Moreover, the results generated by
CNN-based EBD adaptation only (denoted EBDA-CNN) were presented as well,
providing another benchmark for the full ViSTRA2.

Finally, the computational complexity of ViSTRA2 was calculated and normalised
to the corresponding anchor codecs. The encoding process was executed on a
shared cluster, BlueCrystal Phase3 [69] based in the University of Bristol, in
which each CPU node has 16 $\times$2.6 GHz SandyBridge cores and 64GB RAM. The
decoding was run on the GPU nodes of the BlueCrystal Phase 4 [70], each of
which has 14 core 2.4 GHz Intel E5-2680 v4 (Broadwell) CPUs, 138GB of RAM and
two NVIDIA P100 graphic cards.

###  IV-A Compression results

Tables I and II summarise the compression results of ViSTRA2 for JVET CTC test
sequences when integrated with HM 16.20 and VTM 4.01, using PSNR and VMAF as
quality metrics. The rate-quality curves for selected sequences are also shown
in Fig. 3. It can be observed that ViSTRA2 provides consistent bitrate savings
for both host codecs based on PSNR, with average BD-rate values of -12.6% for
HM and -5.5% for VTM. The coding gains are greater when perceptual quality
metric VMAF is employed for quality assessment, and the BD-rate savings (based
on VMAF) are -19.5% and -8.6% for HM and VTM respectively. The coding
improvement from ViSTRA2 for VVC VTM is relatively lower (although consistent)
than that for HEVC HM. This is likely to be due to the significant
enhancements already achieved by VTM over HM.

Comparing to ViSTRA2-w/o CNN, ViSTRA2 is superior in terms of overall coding
efficiency for all test sequences at all resolutions. This improvement is even
higher when VMAF is used to calculate the rate quality performance. It is also
noted that the overall compression performance of EBDA-CNN (without SR
adaptation) is worse than that of the full ViSTRA2, especially for UHD
content, although the former has already been improved from our previous work
[14] due to the much larger training set used. Comparing to the compression
results presented in [10, 17], where the improvement is only evident on high
resolution and low QP cases, ViSTRA2 offers enhanced coding performance across
a wider QP and resolution range. This is due to the more sophisticated network
architecture and more diverse training data employed.

Finally, it can also be observed that ViSTRA2 performs best on UHD sequences
(Class A), where both SR and EBD adaptation were enabled for most tested QP
values. For lower resolutions, SR adaptation was less frequently activated,
and the coding performance of ViSTRA2-CNN is therefore very close (or
identical) to that of EBDA-CNN. Between the two tested QP ranges, ViSTRA2
offers an improved performance for higher QP cases than in low QP scenarios.
This becomes more evident on UHD content (Class A), when spatial resolution
adaptation was more frequently employed.

(a) Campfile-PSNR

(b) DaylightRoad2-PSNR

(c) RitualDance-PSNR

(d) Cactus-PSNR

(e) RaceHorses-PSNR

(f) BasketballPass-PSNR

(g) Campfile-VMAF

(h) DaylightRoad2-VMAF

(i) RitualDance-VMAF

(j) Cactus-VMAF

(k) RaceHorses-VMAF

(l) BasketballPass-VMAF

Figure 3: Rate-PSNR and Rate-VMAF curves for six selected test sequences.

###  IV-B Complexity figures

TABLE III: Computational Complexity of ViSTRA2. Host codec | HM 16.20 | VTM 4.01
---|---|---
Encoder | Decoder | Encoder | Decoder
Class A | 57% | 2,168% | 91% | 1,842%
Class B | 78% | 4,153% | 102% | 3,690%
Class C | 102% | 9,273% | 99% | 7,679%
Class D | 102% | 12,785% | 103% | 11,662%
Average | 80% | 6,421% | 98% | 5,625%

The complexity figures of ViSTRA2 are presented in Table III. It is important
to note the machine learning architecture employed in ViSTRA2 takes account of
the simple filter-based re-sampling process employed in the encoder. This
means that, although we benefit from the reconstruction power of a deep
network, the encoding process complexity remains similar to the reference
encoder. In fact, the average encoding time of ViSTRA2 is shorter (if SR
adaptation is enabled) or equivalent to that of the corresponding anchor
codecs. This is due to the encoder processing down-sampled frames, but also
because we do not employ a neural network at the encoder. During decoding
process, the use of CNN-based up-sampling has significantly increased the
execution time, on average 64 times that of HM and 56 times that of VTM.

##  V Conclusion

A new video coding framework (ViSTRA2) has been presented using CNN-based
spatial resolution and effective bit depth re-sampling. This approach
adaptively reduces the spatial resolution and effective bit depth of input
video content for encoding, and employs a deep CNN at the decoder to
reconstruct its original format. ViSTRA2 has been integrated with HEVC HM
16.20 and VVC VTM 4.01 reference software, showing consistent coding gains for
test sequences at various resolutions based on different quality metrics. It
has further shown that, when deep learning methods are used in an end to end
compression system, there is flexibility in the distribution of complexity
increases and these are not necessarily located at the encoder. Future work
should focus on the complexity optimisation of the CNN to enable more
efficient decoding process.

## Acknowledgment

The authors acknowledge funding from the UK Engineering and Physical Sciences
Research Council (EPSRC, project No. EP/M000885/1), and the NVIDIA GPU Seeding
Grants. The authors would also like to thank Mr Di Ma for his contribution in
developing the large video database for training the employed CNN.

## References

  * [1] ITU-T Rec. H.120, _Codecs for videoconferencing using primary digital group transmission_ , ITU-T Std., 1993.
  * [2] D. R. Bull, _Communicating pictures: a course in image and Video Coding_. Academic Press, 2014.
  * [3] B. Bross, J. Chen, and S. Liu, “Versatile video coding (draft 6),” in _the JVET meeting_ , no. JVET-O2001. ITU-T and ISO/IEC, 2019.
  * [4] D. Liu, Y. Li, J. Lin, H. Li, and F. Wu, “Deep learning-based video coding: A review and a case study,” _arXiv preprint arXiv:1904.12462_ , 2019.
  * [5] S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, and S. Wanga, “Image and video compression with neural networks: A review,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [6] Y. Zhang, S. Kwong, and S. Wang, “Machine learning based video coding optimizations: A survey,” _Information Sciences_ , 2019.
  * [7] G. Sullivan and J.-R. Ohm, “Meeting report of the 10th meeting of the Joint Video Experts Team (JVET),” in _the JVET meeting_ , no. JVET-J_Notes_d6. ITU-T and ISO/IEC, 2019.
  * [8] V.-A. Nguyen, Y.-P. Tan, and W. Lin, “Adaptive downsampling/upsampling for better video compression at low bit rate,” in _2008 IEEE International Symposium on Circuits and Systems_. IEEE, 2008, pp. 1624–1627.
  * [9] G. Georgis, G. Lentaris, and D. Reisis, “Reduced complexity superresolution for low-bitrate video compression,” _IEEE Trans. on Circuits and Systems for Video Technology_ , vol. 26, no. 2, pp. 332–345, 2016.
  * [10] M. Afonso, F. Zhang, and D. R. Bull, “Video compression based on spatio-temporal resolution adaptation,” _IEEE Trans. on Circuits and Systems for Video Technology_ , vol. 29, no. 1, pp. 275–280, January 2019.
  * [11] M. Afonso, F. Zhang, A. Katsenou, D. Agrafiotis, and D. Bull, “Low complexity video coding based on spatial resolution adaptation,” in _Proc. IEEE Int Conf. on Image Processing_. IEEE, 2017\.
  * [12] M. Afonso, F. Zhang, and D. R. Bull, “Spatial resolution adaptation framework for video compression,” in _Proc. SPIE, Applications of Digital Image Processing XLI_ , vol. 10752. International Society for Optics and Photonics, 2018, p. 107520L.
  * [13] D. Ma, M. Afonso, F. Zhang, and D. R. Bull, “Perceptually-inspired super-resolution of compressed videos,” in _Proc. SPIE, Applications of Digital Image Processing XLI_. International Society for Optics and Photonics, 2019.
  * [14] F. Zhang, M. Afonso, and D. R. Bull, “Enhanced video compression based on effective bit depth adaptation,” in _Proc. IEEE Int Conf. on Image Processing_. International Society for Optics and Photonics, 2019.
  * [15] D. Bull, F. Zhang, and M. Afonso, “Video processing method,” United Kingdom Patent GB1 714 791.9, 2017. [Online]. Available: https://www.ipo.gov.uk/p-ipsum/Case/ApplicationNumber/GB1714791.9
  * [16] A. Mackin, M. Afonso, F. Zhang, and D. Bull, “SRQM: A video quality metric for spatial resolution adaptation,” in _Picture Coding Symposium (PCS)_. IEEE, 2018.
  * [17] D. Bull, F. Zhang, and M. Afonso, “Description of SDR video coding technology proposal by University of Bristol (JVET-J0031),” in _the JVET meeting_ , no. JVET-J0031. San Diego, US: ITU-T and ISO/IEC, April 2018.
  * [18] ITU-T Rec. H.264, _Advanced Video Coding for Generic Audiovisual Services_ , ITU-T Std., 2005.
  * [19] ITU-T Rec. H.265, _High efficiency video coding_ , ITU-T Std., 2015.
  * [20] “Alliance for Open Media.” [Online]. Available: https://aomedia.org/
  * [21] AOM. (2019) AOMedia Video 1 (AV1). [Online]. Available: https://github.com/AOMediaCodec
  * [22] K. Choi, J. Chen, D. Rusanovskyy, M. W. Park, C. Roman, J. Samuelsson, X. Li, X. Xu, and K. McCann, “Text of ISO/IEC CD 23094-1, Essential Video Coding.” ISO/IEC JTC1/SC29/WG11, 2019\.
  * [23] Windows Media Codecs. Microsoft. [Online]. Available: https://docs.microsoft.com/en-us/windows/desktop/medfound/windows-media-codecs
  * [24] Dirac. British Broadcasting Corporation R&D. [Online]. Available: http://www.bbc.co.uk/opensource/projects/dirac/
  * [25] Audio and Video Coding Standard. Audio and Video Coding Standard Workgroup. [Online]. Available: http://www.avs.org.cn/english/
  * [26] T. Laude and J. Ostermann, “Deep learning-based intra prediction mode decision for HEVC,” in _Picture Coding Symposium (PCS), 2016_. IEEE, 2016, pp. 1–5.
  * [27] C.-H. Yeh, Z.-T. Zhang, M.-J. Chen, and C.-Y. Lin, “HEVC intra frame coding based on convolutional neural network,” _IEEE Access_ , vol. 6, pp. 50 087–50 095, 2018.
  * [28] J. Li, B. Li, J. Xu, R. Xiong, and W. Gao, “Fully connected network-based intra prediction for image coding,” _IEEE Transactions on Image Processing_ , vol. 27, no. 7, pp. 3236–3247, 2018.
  * [29] J. Pfaff, P. Helle, D. Maniry, S. Kaltenstadler, W. Samek, H. Schwarz, D. Marpe, and T. Wiegand, “Neural network based intra prediction for video coding,” in _Applications of Digital Image Processing XLI_ , vol. 10752\. International Society for Optics and Photonics, 2018, p. 1075213.
  * [30] J. Liu, S. Xia, W. Yang, M. Li, and D. Liu, “One-for-all: Grouped variation network based fractional interpolation in video coding,” _IEEE Transactions on Image Processing_ , 2018.
  * [31] H. Zhang, L. Song, Z. Luo, and X. Yang, “Learning a convolutional neural network for fractional interpolation in HEVC inter coding,” in _Visual Communications and Image Processing (VCIP), 2017 IEEE_. IEEE, 2017, pp. 1–4.
  * [32] Z. Zhao, S. Wang, S. Wang, X. Zhang, S. Ma, and J. Yang, “Enhanced bi-prediction with convolutional neural network for high efficiency video coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2018.
  * [33] N. Yan, D. Liu, H. Li, B. Li, L. Li, and F. Wu, “Convolutional neural network-based fractional-pixel motion compensation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 3, pp. 840–853, 2018\.
  * [34] D. Liu, H. Ma, Z. Xiong, and F. Wu, “CNN-based DCT-like transform for image compression,” in _International Conference on Multimedia Modeling_. Springer, 2018, pp. 61–72.
  * [35] M. M. Alam, T. D. Nguyen, M. T. Hagan, and D. M. Chandler, “A perceptual quantization strategy for hevc based on a convolutional neural network trained on natural images,” in _Applications of Digital Image Processing XXXVIII_ , vol. 9599. International Society for Optics and Photonics, 2015, p. 959918.
  * [36] R. Song, D. Liu, H. Li, and F. Wu, “Neural network-based arithmetic coding of intra prediction modes in HEVC,” in _2017 IEEE Visual Communications and Image Processing (VCIP)_. IEEE, 2017, pp. 1–4.
  * [37] S. Puri, S. Lasserre, and P. Le Callet, “Cnn-based transform index prediction in multiple transforms framework to assist entropy coding,” in _2017 25th European Signal Processing Conference (EUSIPCO)_. IEEE, 2017, pp. 798–802.
  * [38] S. Kuanar, C. Conly, and K. R. Rao, “Deep learning based hevc in-loop filtering for decoder quality enhancement,” in _2018 Picture Coding Symposium (PCS)_. IEEE, 2018, pp. 164–168.
  * [39] W.-S. Park and M. Kim, “CNN-based in-loop filtering for coding efficiency improvement,” in _Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2016 IEEE 12th_. IEEE, 2016, pp. 1–5.
  * [40] R. Yang, M. Xu, T. Liu, Z. Wang, and Z. Guan, “Enhancing quality for hevc compressed videos,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2018.
  * [41] R. Yang, M. Xu, Z. Wang, and T. Li, “Multi-frame quality enhancement for compressed video,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6664–6673.
  * [42] A. Segall, V. Baroncini, J. Boyce, J. Chen, and T. Suzuki, “Joint call for proposals on video compression with capability beyond hevc,” in _the JVET meeting_ , no. JVET-H1002. Macao, CN: ITU-T and ISO/IEC, October 2017.
  * [43] S. Uslubas, E. Maani, and A. Katsaggelos, “A resolution adaptive video compression system,” in _Intelligent Multimedia Communication: Techniques and Applications_. Springer, 2010, pp. 167–194.
  * [44] X. Li, N. Oertel, A. Hutter, and A. Kaup, “Laplace distribution based Lagrangian rate distortion optimization for hybrid video coding,” _IEEE Trans. on Circuits and Systems for Video Technology_ , vol. 19, pp. 193–205, 2008.
  * [45] M. Shen, P. Xue, and C. Wang, “Down-sampling based video coding using super-resolution technique,” _IEEE Trans. on Circuits and Systems for Video Technology_ , vol. 21, no. 6, pp. 755–765, 2011.
  * [46] R.-J. Wang, C.-W. Huang, and P.-C. Chang, “Adaptive downsampling video coding with spatially scalable rate-distortion modeling,” _IEEE Trans. on circuits and systems for video technology_ , vol. 24, no. 11, pp. 1957–1968, 2014\.
  * [47] J. Dong and Y. Ye, “Adaptive downsampling for high-definition video coding,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 24, no. 3, pp. 480–488, 2014.
  * [48] C. Ledig and et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2017, pp. 105–114.
  * [49] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Trans. on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 2, pp. 295–307, 2016.
  * [50] J. Kim, J. K. L., and K. M. L., “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1646–1654.
  * [51] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 136–144.
  * [52] A. Mackin, F. Zhang, and D. Bull, “A study of subjective video quality at various spatial resolutions,” in _Proc. IEEE Int Conf. on Image Processing_. IEEE, 2018.
  * [53] C. G. Bampis, Z. Li, and A. C. Bovik, “Spatiotemporal feature integration and model fusion for full reference video quality assessment,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2018.
  * [54] Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara, “Toward a practical perceptual video quality metric,” _The Netflix Tech Blog_ , 2016\.
  * [55] K. Turkowski, “Filters for common resampling tasks,” in _Graphics gems_. Academic Press Professional, Inc., 1990, pp. 147–165.
  * [56] J. Johnson, A. Alahi, and F.-F. Li, “Perceptual losses for real-time style transfer and super-resolution,” in _European Conference on Computer Vision_. Springer, 2016, pp. 694–711.
  * [57] A. Mackin, F. Zhang, and D. R. Bull, “A study of high frame rate video formats,” _IEEE Trans. on Multimedia_ , vol. 21, no. 6, pp. 1499–1512, June 2019.
  * [58] M. A. Papadopoulos, F. Zhang, D. Agrafiotis, and D. R. Bull, “A video texture database for perceptual compression and quality assessment,” in _Proc. IEEE Int Conf. on Image Processing_. IEEE, 2015.
  * [59] I. Katsavounidis, “NETFLIX - “Chimera” video sequence details and scenes,” November 2015. [Online]. Available: https://www.cdvl.org/documents/NETFLIX_Chimera_4096x2160_Download_Instructions.pdf
  * [60] Harmonic, “Harmonic free 4K demo footage.” [Online]. Available: https://www.harmonicinc.com/free-4k-demo-footage/#4k-clip-center
  * [61] A. Martín et al., “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [62] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations_ , 2015.
  * [63] S. Wenger, A. Segall, M. M. Hannuksela, Hendry, S. McCarthy, Y.-C. Sun, P. Topiwala, and M. Zhou, “JVET AHG report: Layered coding and resolution adaptivity (AHG8),” in _the JVET meeting_ , no. JVET-O0008. ITU-T and ISO/IEC, 2019.
  * [64] F. Bossen, J. Boyce, X. Li, V. Seregin, and K. Sühring, “JVET common test conditions and software reference configurations for SDR video,” in _the JVET meeting_ , no. JVET-M1001. ITU-T and ISO/IEC, 2019.
  * [65] G. Bjøntegaard, “Calculation of average PSNR differences between RD-curves,” in _13th VCEG Meeting_ , no. VCEG-M33. Austin, Texas, USA: ITU-T, April 2001.
  * [66] H. R. Sheikh, A. C. Bovik, and G. de Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” _IEEE Trans. on Image Processing_ , vol. 14, pp. 2117–2128, 2005.
  * [67] S. Li, F. Zhang, L. Ma, and K. H. Ngan, “Image quality assessment by separately evaluating detail losses and additive impairments,” _IEEE Transactions on Multimedia_ , vol. 13, no. 5, pp. 935–949, 2011.
  * [68] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.
  * [69] University of Bristol, “BlueCystal Phase 3.” [Online]. Available: https://www.acrc.bris.ac.uk/acrc/phase3.htm
  * [70] ——, “BlueCystal Phase 4.” [Online]. Available: https://www.acrc.bris.ac.uk/acrc/phase4.htm

|  Fan Zhang (M’12) received the B.Sc. and M.Sc. degrees from Shanghai Jiao
Tong University (2005 and 2008 respectively), and his Ph.D from the University
of Bristol (2012). He is currently working as a Senior Research Associate in
the Visual Information Laboratory, Department of Electrical and Electronic
Engineering, University of Bristol, on projects related to video compression
and machine learning. His research interests include perceptual video
compression, video quality assessment and machine learning based video coding.
---|---
|  Mariana Afonso received the B.S./M.S. degree in Electrical and Computers
Engineering from the University of Porto, Portugal, in 2015 and a PhD in
Electronic and Electrical Engineering from the University of Bristol, UK, in
2019. During her PhD, she was a part of the PROVISION ITN European
Commission’s FP7 Project, a network of leading academic and industrial
organizations in Europe, working on the perceptual video coding. She also
completed a secondment at Netflix, USA, in 2017. She is currently a Research
Scientist in the Video Algorithms team at Netflix. Her research interests
include video compression, video quality assessment, and machine learning.
---|---
|  David R. Bull (M’94-SM’07-F’12) received the B.Sc. degree from the
University of Exeter, Exeter, U.K., in 1980; the M.Sc. degree from University
of Manchester, Manchester, U.K., in 1983; and the Ph.D. degree from the
University of Cardiff, Cardiff, U.K., in 1988. Dr Bull has previously been a
Systems Engineer with Rolls Royce, Bristol, U.K. and a Lecturer at the
University of Wales, Cardiff, U.K. He joined the University of Bristol in 1993
and is currently its Chair of Signal Processing and Director of its Bristol
Vision Institute. In 2001, he co-founded a university spin-off company,
ProVision Communication Technologies Ltd., specializing in wireless video
technology. He has authored over 450 papers on the topics of image and video
communications and analysis for wireless, Internet and broadcast applications,
together with numerous patents, several of which have been exploited
commercially. He has received two IEE Premium awards for his work. He is the
author of three books, and has delivered numerous invited/keynote lectures and
tutorials. Dr. Bull is a fellow of the Institution of Engineering and
Technology.
---|---