# Prediction of Gaussian Volterra Processes with Compound Poisson Jumps

Hamidreza Maleki Almani  Department of Mathematics and Statistics, University
of Vaasa, P.O. Box 700, FIN-65101 Vaasa, FINLAND<EMAIL_ADDRESS>,
Foad Shokrollahi  Department of Mathematics and Statistics, University of
Vaasa, P.O. Box 700, FIN-65101 Vaasa, FINLAND<EMAIL_ADDRESS>and
Tommi Sottinen  Department of Mathematics and Statistics, University of Vaasa,
P.O. Box 700, FIN-65101 Vaasa, FINLAND<EMAIL_ADDRESS>
###### Abstract.

We consider a Gaussian Volterra process with compound Poisson jumps and derive
its prediction law.

###### Key words and phrases:

fractional Brownian motion; Gaussian Volterra process; prediction law;
compound Poisson process

###### 2020 Mathematics Subject Classification:

91G20; 91G80; 60G22

##  1\. Introduction

We are interested in a mixed process $X=G+J$, where the continuous part $G$ is
a so-called Gaussian Volterra process. In older terminology these are
processes that admit canonical representation of multiplicity one. A typical
Gaussian Volterra process is the fractional Brownian motion as shown in Norros
et al. [17]. See Section 4 for the definition of fractional Brownian motion
and for other examples. Intuitively, a Gaussian process $G$ is a Gaussian
Volterra process if one can construct a martingale $M$ from by using a non-
anticipative linear transformation and then represent the original process $G$
in a non-anticipative way as a linear transformation of the martingale $M$.
The motivation to use Gaussian Volterra processes is that for them one can
calculate their prediction law in terms of the kernels that transfer the
Gaussian Volterra process into its driving martingale $M$ and vice versa. In
the mixture $X=G+J$ the jump part $J$ will be an independent compound Poisson
process with square-integrable jump distribution.

One motivation to study processes of the type $X=G+J$ comes from mathematical
finance. Indeed, it is well-known that the returns of financial assets do not
follow Gaussian distribution [7, 12, 16] and the returns also exhibit jumps,
or shocks [1, 2, 6, 14, 18]. Also, there is evidence of long-range dependence
in the returns also explain the presence of long-memory [5, 9, 13, 15, 19].
Thus models where the returns are Gaussian with jumps seems more reasonable:
The Gaussian part could take care of the long-range dependence with fractional
Brownian motion (fBm) as the Gaussian Volterra process, and the shocks would
come from the compound Poisson part. Then one can use the result of this paper
to calculate imperfect hedges in the mixed model in the similar way as done in
[20, 22]. Indeed, this is work in progress by the authors.

In this paper we derive the prediction law of the mixed process $X=G+J$.

The rest of the paper is organized as follows. In Section 2 we define Gaussian
Volterra processes and derive their prediction laws. Section 3 is the main
section of the paper where we introduce the Gaussian Volterra processes with
compound Poisson jumps and derive their prediction laws. Finally, in Section 4
we provide examples of Gaussian Volterra processes.

##  2\. Gaussian Volterra processes

A Gaussian Volterra process is a Gaussian process that has a canonical
representation of multiplicity one with respect to a Gaussian martingale. The
Gaussian Volterra process is defined in Definition 2.1 below in terms of
covariance functions. The Gaussian Volterra representations follow from
Definition 2.1 and are stated in Proposition 2.1.

For convenience we consider processes over the compact time-interval $[0,T]$
with an arbitrary but fixed time horizon $T>0$.

Let $G=(G_{t})_{t\in[0,T]}$ be a centered Gaussian process with $G_{0}=0$ and
covariance function $R\colon[0,T]^{2}\to\mathbb{R}$ defined on a complete
probability space $(\Omega,\mathscr{F},\mathbb{P})$.

A kernel $K:[0,T]^{2}\to\mathbb{R}$ is a Volterra kernel if $K(t,s)=0$
whenever $t<s$. For a Volterra kernel $K$ we define its associated operator
$\mathrm{K}$ as.

$\mathrm{K}[f](t)=\int_{0}^{t}f(s)K(t,s)\,\mathrm{d}s.$

Denote

$\displaystyle\mathbf{1}_{t}(s)$ $\displaystyle=$
$\displaystyle\mathbf{1}_{[0,t)}(s)=\left\\{\begin{array}[]{rl}1,&\mbox{if
}s\in[0,t),\\\ 0,&\mbox{otherwise}\end{array}\right..$

The adjoint associated operator $\mathrm{K}^{*}$ of the Volterra kernel $K$ is
given by extending linearly the relation

$\mathrm{K}^{*}[\mathbf{1}_{t}](s)=K(t,s).$

It turns out that $\mathrm{K}^{*}_{G}$ for a Gaussian Volterra process with
covariance

$R(t,s)=\int_{0}^{t\wedge s}K(t,u)K(s,u)\,\mathrm{d}v(u)$

extends to an isometry from $\Lambda$ to $L^{2}([0,T],\mathrm{d}v)$ where $v$
is given in Definition 2.1(i) and $\Lambda$, the space of wiener integrands,
is the closure of the indicator functions $\mathbf{1}_{t}$, $t\in[0,T]$, in
the inner product

${\langle\mathbf{1}_{t},\mathbf{1}_{s}\rangle}_{\Lambda}=R(t,s).$

######  Remark 2.1.

By Aòs et al. [3], if $K$ is of bounded variation in its first argument, we
can write for any elementary $f$

$\mathrm{K}^{*}[f](t)=f(t)K(T,t)-\int_{t}^{T}\left[f(u)-f(t)\right]K(\mathrm{d}u,t).$

Moreover, we have for elementary $f$ and $g$ that

$\int_{0}^{T}\mathrm{K}^{*}[f](t)g(t)\,\mathrm{d}t=\int_{0}^{T}g(t)\mathrm{K}[h](\mathrm{d}t)$

justifying the name “adjoint” associated operator.

For Gaussian Volterra representations we recall what is the co-called abstract
Wiener integral (for more information on abstact Wiener integrals and their
relation to conditioning we refer to [23])).

The linear space $\mathscr{L}$ is the closure of of the random variables
$G_{t}$, $t\in[0,T]$, in $L^{2}(\Omega,\mathscr{F},\mathbb{P})$. The spaces
$\Lambda$ and $\mathscr{L}$ are isometric. Indeed, the mapping

$\mathbf{1}_{t}\mapsto G_{t}$

extends to an isometry. This isometry is called the abstract Wiener integral
and we denote it

$\int_{0}^{T}f(t)\,\mathrm{d}G_{t}$

for a $f\in\Lambda$.

######  Definition 2.1 (Gaussian Volterra process).

Let $G=(G_{t})_{t\in[0,T]}$ be a centered Gaussian process with covariance
function $R\colon[0,\infty)^{2}\to\mathbb{R}$. Assume that

  1. (i)

there exists an increasing function $v\colon[0,T]\to\mathbb{R}$ and a Volterra
kernel $K\colon[0,T]^{2}\to\mathbb{R}$ such that
$\int_{0}^{t}K(t,s)^{2}\,\mathrm{d}v(s)<\infty$ for all $t\in[0,T]$ and

$R(t,s)=\int_{0}^{t\wedge s}K(t,u)K(s,u)\,\mathrm{d}v(u),$

  2. (ii)

for each $t\in[0,T]$ the equation

$\mathrm{K}^{*}[K^{-1}(t,\cdot)](s)=\mathbf{1}_{t}(s)$

admits a solution $K^{-1}(t,\cdot)$.

Note that by Definition 2.1(ii) the operator $\mathrm{K}^{*}$ is invertible
and we have

$(\mathrm{K}^{*})^{-1}[\mathbf{1}_{t}](s)=K^{-1}(t,s).$

We note that due to Definition 2.1(i) for a Gaussian Volterra process the
space $\Lambda$ is isometric to $L^{2}([0,T],\mathrm{d}v)$. Indeed, we have

${\langle
f,g\rangle}_{\Lambda}={\langle\mathrm{K}^{*}[f],\mathrm{K}^{*}[g]\rangle}_{L^{2}([0,T],\mathrm{d}v)}.$

In particular, this means that the mapping $\mathrm{K}^{*}$ in Definition
2.1(ii) is an isometry between $\Lambda$ and $L^{2}([0,t],\mathrm{d}v)$. We
also note that due to Definition 2.1(ii) the operator $\mathrm{K}^{*}$ is
invertible and we have, in particular, that

$(\mathrm{K}^{*})^{-1}[\mathbf{1}_{t}](s)=K^{-1}(t,s).$

The following representation proposition is a direct consequence of Definition
2.1. Indeed, Proposition 2.1 could have been taken as the definition of
Gaussian Volterra process.

######  Proposition 2.1 (Volterra representation).

Let $G$ be a Gaussian Volterra process. Let $K^{-1}$ be the kernel in
Definition 2.1(ii). Then the process

$M_{t}=\int_{0}^{t}K^{-1}(t,s)\,\mathrm{d}G_{s}$

is a Gaussian martingale with bracket $\langle M\rangle_{t}=v(t)$. Moreover,

$G_{t}=\int_{0}^{t}K(t,s)\,\mathrm{d}M_{s},$

where $v$ and $K$ are as in Definition 2.1(i).

Note that form Proposition 2.1 we immediately see that the filtrations
$\mathbb{F}^{G}$ and $\mathbb{F}^{M}$ coincide.

The Volterra representations of Proposition 2.1 extend immediately to the
following transfer principle for Wiener integrals.

######  Proposition 2.2 (Transfer principle).

Let $f\in\Lambda$ and $g\in L^{2}([0,T],\mathrm{d}v)$. Then

$\displaystyle\int_{0}^{T}f(t)\,\mathrm{d}G_{t}$ $\displaystyle=$
$\displaystyle\int_{0}^{T}\mathrm{K}^{*}[f](t)\,\mathrm{d}M_{t},$
$\displaystyle\int_{0}^{T}g(t)\,\mathrm{d}M_{t}$ $\displaystyle=$
$\displaystyle\int_{0}^{T}(\mathrm{K}^{*})^{-1}[g](t)\,\mathrm{d}G_{t}.$

In what follows we will use the following notation for the conditional mean,
the conditional covariance and the conditional law of a stochastic process
$Y$:

$\displaystyle\hat{m}_{t}^{Y}(u)$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[Y_{t}\big{|}\mathscr{F}^{Y}_{u}\right],$
$\displaystyle\hat{R}_{Y}(t,s|u)$ $\displaystyle=$
$\displaystyle\mathbb{C}\mathrm{ov}[Y_{t},Y_{s}\big{|}\mathscr{F}^{Y}_{u}],$
$\displaystyle\hat{P}_{t}^{Y}(\mathrm{d}y|u)$ $\displaystyle=$
$\displaystyle\mathbb{P}\left[Y_{t}\in\mathrm{d}y\big{|}\mathscr{F}^{Y}_{u}\right]$

We end this section by stating the prediction formula for Gaussian Volterra
processes. The formula and its proof is similar to that given in [21]. We give
here the proof in detail for the convenience of the readers.

######  Proposition 2.3 (Volterra Prediction).

Let $G$ be a Gaussian Volterra process as in Definition 2.1. Let $u\leq s\leq
t\leq T$. Denote

$\Psi(t,s|u)=(\mathrm{K}^{*})^{-1}[K(t,\cdot)-K(u,\cdot)](s)$

and

$\Phi(\mathrm{d}x;\mu,\sigma^{2})=\frac{1}{\sqrt{2\pi\sigma^{2}}}\mathrm{e}^{-\frac{1}{2}\frac{(x-\mu)^{2}}{\sigma^{2}}}\,\mathrm{d}x.$

Then

$\displaystyle\hat{m}^{G}_{t}(u)$ $\displaystyle=$ $\displaystyle
G_{u}-\int_{0}^{u}\Psi(t,s|u)\,\mathrm{d}G_{u},$
$\displaystyle\hat{R}_{G}(t,s|u)$ $\displaystyle=$ $\displaystyle
R(t,s)-\int_{0}^{u}K(t,x)K(s,x)\,\mathrm{d}v(x),$
$\displaystyle\hat{P}_{t}^{G}(\mathrm{d}x|u)$ $\displaystyle=$
$\displaystyle\Phi(\mathrm{d}x;\hat{m}_{t}^{G}(u),\hat{R}(t,t|u)).$

###### Proof.

It is well-known that conditional Gaussian processes are Gaussian. Therefore
it is enough to identify the conditional mean and conditional covariance.

We consider first the conditional mean. Now

$\displaystyle\hat{m}^{G}_{t}(u)$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[G_{t}\big{|}\mathscr{F}_{u}^{G}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\int_{0}^{t}K(t,s)\,\mathrm{d}M_{s}\big{|}\mathscr{F}_{u}^{M}\right]$
$\displaystyle=$ $\displaystyle\int_{0}^{u}K(t,s)\,\mathrm{d}M_{s}$
$\displaystyle=$
$\displaystyle\int_{0}^{u}K(u,s)\,\mathrm{d}M_{s}+\int_{0}^{u}\left[K(t,s)-K(u,s)\right]\,\mathrm{d}M_{s}$
$\displaystyle=$ $\displaystyle
G_{u}-\int_{0}^{u}(\mathrm{K}^{*})^{-1}\left[K(t,\cdot)-K(u,\cdot)\right](s)\,\mathrm{d}G_{s}.$

Let us then consider the conditional covariance. Now

$\displaystyle\hat{R}_{G}(t,s|u)$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\left(G_{t}-\hat{m}^{G}_{t}(u)\right)\left(G_{s}-\hat{m}^{G}_{s}(u)\right)\big{|}\mathscr{F}^{G}_{u}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\left(\int_{0}^{t}K(t,x)\,\mathrm{d}M_{x}-\hat{m}^{G}_{t}(u)\right)\left(\int_{0}^{s}K(s,x)\,\mathrm{d}M_{x}-\hat{m}^{G}_{s}(u)\right)\bigg{|}\mathscr{F}^{M}_{u}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\int_{u}^{t}K(t,x)\,\mathrm{d}M_{x}\int_{u}^{s}K(s,x)\,\mathrm{d}M_{x}\bigg{|}\mathscr{F}^{M}_{u}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\int_{u}^{t}K(t,x)\,\mathrm{d}M_{x}\int_{u}^{s}K(s,x)\,\mathrm{d}M_{x}\right]$
$\displaystyle=$ $\displaystyle\int_{u}^{t\wedge
s}K(t,x)K(s,x)\,\mathrm{d}v(x)$ $\displaystyle=$
$\displaystyle\int_{0}^{t\wedge
s}K(t,x)K(s,x)\,\mathrm{d}v(x)-\int_{0}^{u}K(t,x)K(s,x)\,\mathrm{d}v(x)$
$\displaystyle=$ $\displaystyle
R(t,s)-\int_{0}^{u}K(t,x)K(s,x)\,\mathrm{d}v(x).$

∎

##  3\. Gaussian Volterra process with jumps

In this section we prove our main result, the prediction formula for Gaussian
Volterra processes with compound Poisson jumps. Indeed, we consider the
process $X=(X_{t})_{t\in[0,T]}$ given by

(3.1) $X=G+J,$

where $G$ is a continuous Gaussian Volterra process and $J$ is an independent
compound Poisson process with intensity $\lambda$ and jump distribution $F$.
In other words

$J_{t}=\sum_{k=1}^{N_{t}}\xi_{k},$

where $N=(N_{t})_{t\in[0,T]}$ is a Poisson process with intensity $\lambda$
and the jumps $\xi_{k}$, $k\in\mathbb{N}$, are i.i.d. with common distribution
$F$, and they are independent of the Poisson process $N$ and the Gaussian
Volterra process $G$. We denote

$\displaystyle\mu_{1}$ $\displaystyle=$ $\displaystyle\mathbb{E}[\xi_{k}],$
$\displaystyle\mu_{2}$ $\displaystyle=$
$\displaystyle\mathbb{E}[\xi_{k}^{2}].$

We note that it is crucial that the Gaussian Volterra part is continuous since
it implies that $\mathbb{F}^{X}=\mathbb{F}^{G,J}$, i.e. the signals $G$ and
$J$ can be separated from the signal $X$. We refer to [4] and references
therein on the continuity of Gaussian processes.

Our main theorem is the following.

######  Theorem 3.1 (Mixed Prediction).

Let $X$ be given by (3.1). Then

$\displaystyle\hat{m}^{X}_{t}(u)$ $\displaystyle=$ $\displaystyle
X_{u}-\int_{0}^{u}\Psi(t,s|u)\,\mathrm{d}G_{s}+\lambda(t-u)\mu_{1},$
$\displaystyle\hat{R}_{X}(t,s|u)$ $\displaystyle=$ $\displaystyle
R(t,s)-\int_{0}^{u}K(t,x)K(s,x)\,\mathrm{d}v(x)+\lambda(t\wedge s-u)\mu_{2},$
$\displaystyle\hat{P}^{X}_{t}(\mathrm{d}x|u)$ $\displaystyle=$
$\displaystyle\int_{y\in\mathbb{R}}\Phi\left(\mathrm{d}x-y;\hat{m}^{G}_{t}(u),\hat{R}_{G}(t,t|u)\right)\sum_{n=0}^{\infty}\frac{\mathrm{e}^{-\lambda(t-u)}(\lambda(t-u))^{n}}{n!}F^{*n}\left(\mathrm{d}y-J_{u}\right).$

Here $F^{*n}$ is the $n$-fold convolution of the distribution $F$:

$\displaystyle F^{*1}(\mathrm{d}x)$ $\displaystyle=$ $\displaystyle
F(\mathrm{d}x),$ $\displaystyle F^{*n}(\mathrm{d}x)$ $\displaystyle=$
$\displaystyle\int_{y\in\mathbb{R}}F(\mathrm{d}x-y)F^{*(n-1)}(\mathrm{d}y).$

###### Proof.

Let us begin with the mean $\hat{m}^{X}_{t}(u)$. The conditional mean of $G$
is already known. As for the conditional mean of $J$, we have, by
independence, that

$\displaystyle\hat{m}^{J}_{t}(u)$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[J_{t}\big{|}\mathscr{F}^{J}_{u}\right]$
$\displaystyle=$ $\displaystyle
J_{u}+\mathbb{E}\left[J_{t}-J_{u}\big{|}\mathscr{F}^{J}_{u}\right]$
$\displaystyle=$ $\displaystyle J_{u}+\mathbb{E}\left[J_{t-u}\right]$
$\displaystyle=$ $\displaystyle J_{u}+\lambda(t-u)\mu_{1}.$

The formula for the conditional mean follows from this.

Let us then consider the conditional variance $\hat{R}_{X}(t,s|u)$. By
independence we have

$\hat{R}_{X}(t,s|u)=\hat{R}_{G}(t,s|u)+\hat{R}_{J}(t,s|u).$

Now $\hat{R}_{G}(t,s|u)$ is known and for $\hat{R}_{J}(t,s|u)$ we have

$\displaystyle\hat{R}_{J}(t,s|u)$ $\displaystyle=$
$\displaystyle\mathbb{C}\mathrm{ov}\left[J_{t},J_{s}\big{|}\mathscr{F}^{J}_{u}\right]$
$\displaystyle=$
$\displaystyle\mathbb{C}\mathrm{ov}\left[J_{t}-J_{u},J_{s}-J_{u}\big{|}\mathscr{F}^{J}_{u}\right]$
$\displaystyle=$
$\displaystyle\mathbb{C}\mathrm{ov}\left[J_{t-u},J_{s-u}\right]$
$\displaystyle=$ $\displaystyle\lambda(t\wedge s-u)\mu_{2}.$

Finally, let us consider the conditional law $\hat{P}^{X}_{t}(\mathrm{d}x|u)$.
By the law of total probability and independence we have

$\displaystyle\hat{P}_{t}(\mathrm{d}x|u)$ $\displaystyle=$
$\displaystyle\int_{y\in\mathbb{R}}\mathbb{P}\left[G_{t}\in\mathrm{d}x-y\big{|}J_{t}=y,\mathscr{F}_{u}^{X}\right]\mathbb{P}\left[J_{t}\in\mathrm{d}y\big{|}\mathscr{F}_{u}^{X}\right]$
$\displaystyle=$
$\displaystyle\int_{y\in\mathbb{R}}\mathbb{P}\left[G_{t}\in\mathrm{d}x-y\big{|}\mathscr{F}_{u}^{G}\right]\mathbb{P}\left[J_{t}\in\mathrm{d}y\big{|}\mathscr{F}_{u}^{J}\right]$
$\displaystyle=$
$\displaystyle\int_{y\in\mathbb{R}}\hat{P}^{G}_{t}(\mathrm{d}x-y|u)\mathbb{P}\left[J_{t}-J_{u}\in\mathrm{d}y-J_{u}\big{|}J_{u}\right]$

and

$\displaystyle\mathbb{P}\left[J_{t}-J_{u}\in\mathrm{d}y-J_{u}\big{|}J_{u}\right]$
$\displaystyle=$
$\displaystyle\sum_{n=0}^{\infty}\mathbb{P}\left[J_{t}-J_{u}\in\mathrm{d}y-J_{u}\big{|}N_{t}-N_{u}=n,J_{u}\right]\mathbb{P}\left[N_{t}-N_{u}=n\big{|}J_{u}\right]$
$\displaystyle=$
$\displaystyle\sum_{n=0}^{\infty}\mathbb{P}\left[\sum_{k=1}^{n}\xi_{k}\in\mathrm{d}y-J_{u}\Bigg{|}J_{u}\right]\mathbb{P}[N_{t-u}=n]$
$\displaystyle=$
$\displaystyle\sum_{n=0}^{\infty}F^{*n}\left(\mathrm{d}y-J_{u}\right)\mathbb{P}[N_{t-u}=n]$

The formula for the conditional law follows from this by plugging in
$\hat{P}^{G}_{t}(\mathrm{d}x|u)$ and the Poisson probabilities. ∎

######  Remark 3.1.

It is interesting to note that just like in the Gaussian case, also in the
mixed case, the conditional covariance is deterministic.

##  4\. Examples

In this section we give examples for different Gaussian Volterra processes $G$
for the prediction formula of Theorem 3.1. This means that we give the kernel
$K$, the function $v$, and the kernel $\Psi$.

######  Example 4.1 (fBm).

The fractional Brownian motion $B^{H}$ with Hurst index $H\in(0,1)$ is a
centered Gaussian process with covariance

$R_{H}(t,s)=\frac{1}{2}\left(t^{2H}+s^{2H}-|t-s|^{2H}\right).$

By Norros et al. [17] (see also [21])) $B^{H}$ is a Gaussian Volterra process
with $v(t)=t$ and the Volterra kernel

(4.1)
$K_{H}(t,s)=c_{H}\Bigg{\\{}\left(\frac{t}{s}\right)^{H-\frac{1}{2}}(t-s)^{H-\frac{1}{2}}\,-\,(H-\frac{1}{2})s^{\frac{1}{2}-H}\int_{s}^{t}u^{H-\frac{3}{2}}(u-s)^{H-\frac{1}{2}}\,\mathrm{d}u\Bigg{\\}},$

with a normalizing constant

$c_{H}=\sqrt{\frac{2H\Gamma(\frac{3}{2}-H)}{\Gamma(\frac{1}{2}+H)\Gamma(2-2H)}},$

and we have

(4.2)
$\Psi_{H}(t,s|u)=\frac{\sin(\pi(H-\frac{1}{2}))}{\pi}s^{\frac{1}{2}-H}(u-s)^{\frac{1}{2}-H}\int_{u}^{t}\frac{z^{H-\frac{1}{2}}(z-u)^{H-\frac{1}{2}}}{z-s}\,\mathrm{d}z.$

######  Example 4.2 (ccmfBm).

The long-range dependent completely correlated mixed fractional Brownian
motion (ccmfBm) was introduced in [11]. It is a Gaussian process

$M=aW+bB^{H}$

where $B^{H}$ is a fractional Brownian motion with $H>1/2$, and it is
constructed from the Brownnian motion $W$ by using the Volterra representation

$B^{H}_{t}=\int_{0}^{t}K_{H}(t,s)\,\mathrm{d}W_{s}$

(see Example 4.1 above). The ccfBm, $M$ is also a Gaussian Volterra process
with $v(t)=t$ and the Volterra kernel

$K_{a,b,H}(t,s)=a\,\mathbf{1}_{t}(s)+b\,K_{H}(t,s).$

In other word, it has the following Volterra representation

$M_{t}=\int_{0}^{t}K_{a,b,H}(t,s)\,\mathrm{d}W_{s},$

and so

$\Psi_{a,b,H}(t,s|u)=(\mathrm{K}_{a,b,H}^{*})^{-1}\left[K_{a,b,H}(t,\cdot)-K_{a,b,H}(u,\cdot)\right](s).$

Here

$\displaystyle\mathrm{K}_{a,b,H}^{*}[f](t)=af(t)+\frac{b\,c(H)(H-\frac{1}{2})}{t^{H-\frac{1}{2}}}\int_{t}^{T}f(u)\frac{u^{H-\frac{1}{2}}}{(u-t)^{\frac{3}{2}-H}}\,\mathrm{d}u,$

for all $f\in\Lambda_{a,b,H}$, the space of all integrands from $M$, which is
simply $L^{2}[0,T]$ in this case. The $(\mathrm{K}_{a,b,H}^{*})^{-1}$ is the
inverse operator of $\mathrm{K}_{a,b,H}^{*}$, where from [11]

$\displaystyle(\mathrm{K}_{a,b,H}^{*})^{-1}[f](t)=f(T)K_{a,b,H}^{-1}(t,T)-\int_{t}^{T}f(s)K_{a,b,H}^{-1}(\mathrm{d}s,t),$
$\displaystyle
K_{a,b,H}^{-1}(t,s)=\frac{1}{a}\mathbf{1}_{t}(s)+\frac{1}{a}\sum_{k=1}^{\infty}(-1)^{k}\left(\frac{b}{a}\right)^{k}\gamma_{k}(t,s),$
$\displaystyle\gamma_{k}(t,s)=\frac{c(H)^{k}\Gamma(H+\frac{1}{2})^{k}}{\Gamma(k(H-\frac{1}{2}))}\frac{1}{s^{H-\frac{1}{2}}}\int_{s}^{t}u^{H-\frac{1}{2}}(u-s)^{k(H-\frac{1}{2})-1}\mathrm{d}u.$

######  Example 4.3 (mfBm).

The mixed fractional Brownian motion (mfBm) is the process

$\tilde{M}=W+B^{H}$

where the Brownian motion $W$ and the fractional Brownian motion $B^{H}$ are
independent. The mfBm was introduced by Cheridito [10]. In Cai et al. [8] it
was shown that the mfBm is a Gaussian Volterra process with $v(t)=t$ and a
certain kernel $\tilde{K}_{H}$.

Indeed, let $H>1/2$ and let $L(t,s)$ be the solution of the equation

$L(t,s)+H(2H-1)\int_{0}^{t}L(t,x)|s-x|^{2H-2}\,\mathrm{d}x=-H(2H-1)|t-s|^{2H-2},\quad
0\leq s,t$

Then by Theorem 2.2 of [8] we have the following: denote

$\phi(t)=1-\int_{0}^{t}L(t,x)\,\mathrm{d}x.$

Then

$\tilde{W}_{t}=\mathbb{E}\left[\int_{0}^{t}\phi(s)\,\mathrm{d}W_{s}\,\bigg{|}\,\mathscr{F}^{\tilde{M}}_{t}\right]=\int_{0}^{t}q(t,s)\,\mathrm{d}\tilde{M}_{s},$

is a Brownian motion, where $q(t,s)$ is the unique solution of the Wiener–Hopf
equation:

$q(t,s)+H(2H-1)\int_{0}^{t}q(t,x)|s-x|^{2H-2}\,\mathrm{d}x=\phi(s),\quad 0\leq
s,t$

and

$\tilde{M}_{t}=\int_{0}^{t}\tilde{K}_{H}(t,s)\,\mathrm{d}\tilde{W}_{s},$

where

$\tilde{K}_{H}(t,s)=-\frac{\partial}{\partial
s}\int_{s}^{t}q(t,x)\mathrm{d}x.$

and here we have

$\tilde{\Psi}(t,s|u)=(\widetilde{\mathrm{K}}_{H}^{*})^{-1}\left[\tilde{K}_{H}(t,\cdot)-\tilde{K}_{H}(u,\cdot)\right](s),$

where

$\widetilde{\mathrm{K}}_{H}^{*}[f](t)=f(t)\tilde{K}_{H}(T,t)-\int_{t}^{T}\left[f(u)-f(t)\right]\tilde{K}_{H}(\mathrm{d}u,t),$

for all $f\in\tilde{\Lambda}$, the space of all integrands from $\tilde{M}$,
and $(\widetilde{\mathrm{K}}_{H}^{*})^{-1}$ is the inverse operator of
$\widetilde{\mathrm{K}}_{H}^{*}$.

## References

  * [1] V. Akgiray and G. Booth, Stock price processes with discontinuous time paths: An empirical examination, Financial Review, 21 (1986), pp. 163–184.
  * [2] V. Akgiray and G. G. Booth, Compound distribution models of stock returns: An empirical comparison, Journal of Financial Research, 10 (1987), pp. 269–280.
  * [3] E. Alòs, O. Mazet, and D. Nualart, Stochastic calculus with respect to Gaussian processes, Ann. Probab., 29 (2001), pp. 766–801.
  * [4] E. Azmoodeh, T. Sottinen, L. Viitasaari, and A. Yazigi, Necessary and sufficient conditions for Hölder continuity of Gaussian processes, Statist. Probab. Lett., 94 (2014), pp. 230–235.
  * [5] R. T. Baillie, Long memory processes and fractional integration in econometrics, Journal of econometrics, 73 (1996), pp. 5–59.
  * [6] C. A. Ball and W. N. Torous, On jumps in common stock prices and their impact on call option pricing, The Journal of Finance, 40 (1985), pp. 155–173.
  * [7] R. C. Blattberg and N. J. Gonedes, A comparison of the stable and student distributions as statistical models for stock prices, in Perspectives on promotion and database marketing: The collected works of Robert C Blattberg, World Scientific, 2010, pp. 25–61.
  * [8] C. Cai, P. Chigansky, and M. Kleptsyna, Mixed Gaussian processes: A filtering approach, Ann. Probab., 44 (2016), pp. 3032–3075.
  * [9] K. Chan and A. Hameed, Stock price synchronicity and analyst coverage in emerging markets, Journal of Financial Economics, 80 (2006), pp. 115–147.
  * [10] P. Cheridito, Mixed fractional Brownian motion, Bernoulli, 7 (2001), pp. 913–934.
  * [11] J. Dufitinema, F. Shokrollahi, T. Sottinen, and L. Viitasaari, Long-range dependent completely correlated mixed fractional brownian motion, arXiv:2104.04992, (2021).
  * [12] E. F. Fama, The behavior of stock-market prices, The journal of Business, 38 (1965), pp. 34–105.
  * [13] C. R. Harvey, Predictable risk and returns in emerging markets, The review of financial studies, 8 (1995), pp. 773–816.
  * [14] R. A. Jarrow and E. R. Rosenfeld, Jump risks and the intertemporal capital asset pricing model, Journal of Business, (1984), pp. 337–351.
  * [15] S.-J. Kim and E. Wu, Sovereign credit ratings, capital flows and financial sector development in emerging markets, Emerging markets review, 9 (2008), pp. 17–39.
  * [16] B. B. Mandelbrot and B. B. Mandelbrot, The variation of certain speculative prices, Springer, 1997.
  * [17] I. Norros, E. Valkeila, and J. Virtamo, An elementary approach to a Girsanov formula and other analytical results on fractional Brownian motions, Bernoulli, 5 (1999), pp. 571–587.
  * [18] S. J. Press, A compound events model for security prices, Journal of business, (1967), pp. 317–335.
  * [19] R. G. Rajan and L. Zingales, The great reversals: the politics of financial development in the twentieth century, Journal of financial economics, 69 (2003), pp. 5–50.
  * [20] F. Shokrollahi and T. Sottinen, Hedging in fractional Black-Scholes model with transaction costs, Statist. Probab. Lett., 130 (2017), pp. 85–91.
  * [21] T. Sottinen and L. Viitasaari, Prediction law of fractional Brownian motion, Statist. Probab. Lett., 129 (2017), pp. 155–166.
  * [22] , Conditional-mean hedging under transaction costs in Gaussian models, Int. J. Theor. Appl. Finance, 21 (2018), pp. 1850015, 15.
  * [23] T. Sottinen and A. Yazigi, Generalized Gaussian bridges, Stochastic Process. Appl., 124 (2014), pp. 3084–3105.


# ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images

Weiqi Li∗1,2, Shijie Zhao†2, Bin Chen1, Xinhua Cheng1, Junlin Li2, Li Zhang2,
Jian Zhang†1
1Peking University 2ByteDance Inc<EMAIL_ADDRESS>
(2024)

###### Abstract.

With the advent of virtual reality technology, omnidirectional image (ODI)
rescaling techniques are increasingly embraced for reducing transmitted and
stored file sizes while preserving high image quality. Despite this progress,
current ODI rescaling methods predominantly focus on enhancing the quality of
images in equirectangular projection (ERP) format, which overlooks the fact
that the content viewed on head mounted displays (HMDs) is actually a rendered
viewport instead of an ERP image. In this work, we emphasize that focusing
solely on ERP quality results in inferior viewport visual experiences for
users. Thus, we propose ResVR, which is the first comprehensive framework for
the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR
ERP images for transmission while rendering high-quality viewports for users
to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is
developed to tackle the complex mapping between the viewport and ERP, enabling
end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape
representation technique is innovatively derived from spherical
differentiation to significantly improve the visual quality of rendered
viewports. Extensive experiments demonstrate that our ResVR outperforms
existing methods in viewport rendering tasks across different fields of view,
resolutions, and view directions while keeping a low transmission
overhead111The complete code and pre-trained models of our method will be made
available.11footnotetext: ∗ Interns in MMLab, ByteDance.11footnotetext: †
Corresponding author..

Omnidirectional Image, Image Rescaling, Viewport Rendering

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††isbn:
978-1-4503-XXXX-X/18/06††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing
methodologies Computer vision

Figure 1. Proposed ResVR compared to previous ODI rescaling and viewport
rendering paradigms. (a) Conventional methods focus on improving the quality
of rescaled ERP images, resulting in inferior visual experiences. (b)
Considering the fact that the desired content viewed on HMDs is a rendered
viewport instead of an ERP image, our ResVR directly optimizes the quality of
the final viewport for users through a novel discrete pixel sampling strategy
and a spherical pixel shape representation technique. (c) Visual and PSNR
comparisons of rendered viewports between the two pipelines in (a) and (b) on
user HMDs.

##  1\. Introduction

With the growing interest in virtual reality and augmented reality,
omnidirectional images (ODIs), also referred to as 360° or panoramic images,
attract great attention within the computer vision community for their
immersive and interactive capabilities. Although ODIs can capture scenes
across a comprehensive 360°$\times$180° views, head-mounted displays (HMDs)
often present a limited field-of-view (FoV), necessitating resolutions as high
as 4K$\times$8K (Ai et al., 2022) to preserve details in a small viewport.
High-resolution (HR) ODIs are typically stored on cloud servers by platforms
of virtual reality media, requiring real-time download by users. This can
degrade the visual experience of users, particularly under poor internet
conditions.

Image rescaling (Xiao et al., 2020; Liang et al., 2021b; Xiao et al., 2023;
Cheng et al., 2021; Xing et al., 2021; Xie et al., 2021) emerges as an
effective method to reduce image file size for storage and transmission while
preserving high quality in the reconstructed images at the user end. This
technique first downscales HR images to low-resolution (LR) ones that keep the
most important visual details, then upscales them back to their original HR
versions. It not only minimizes the file size of LR images but also maintains
the quality of reconstructed HR images (Qi et al., 2023), thus becoming a
straightforward approach for efficient transmission and storage for ODIs from
cloud servers to user HMDs. However, the prevalent storage and transmission
format for ODIs, i.e., the equirectangular projection (ERP), has directed
state-of-the-art ODI rescaling works (Guo et al., 2023) to focus on enhancing
the quality of ERP images. As depicted in Fig. 1 (a), after receiving an LR
ODI, the typical process on HMDs is two-step, involving (1) upscaling HR ODIs
in ERP format from LR images ($\hat{\mathbf{I}}^{\text{LR-
ERP}}\mapsto\hat{\mathbf{I}}^{\text{HR-ERP}}$), and (2) projecting them onto
the viewport ($\hat{\mathbf{I}}^{\text{HR-
ERP}}\mapsto\hat{\mathbf{I}}^{\text{HR-View}}$) using traditional
interpolation methods such as bilinear. This pipeline does not fully account
for the ultimate rendered image of the HMD viewport
$\hat{\mathbf{I}}^{\text{HR-View}}$, particularly lacking optimization for the
final viewing experience. As Fig. 1 (c) shows, the quality of images seen by
the user on HMDs can be significantly lower than anticipated.

In this paper, we point out that (1) the content viewed on HMDs is actually a
rendered viewport, not an ERP image, and (2) focusing solely on the quality of
ERP images will result in sub-optimal viewport visual experiences. To improve
the viewing experience for users, there is a need to develop a comprehensive
solution that is optimized for end-to-end ODI processing from the storage of
ERP images to the display of the final viewport. To this end, we propose
ResVR, a novel framework for joint Rescaling and Viewport Rendering of ODIs,
marking an innovative step towards comprehensive end-to-end ODI processing. As
shown in Fig. 1 (b), ResVR aligns the optimization of network parameters with
our primary goal of improving the quality of the final viewport. By utilizing
such a new methodology, the viewports rendered through our ResVR framework
exhibit enhanced details and fewer artifacts compared to those produced by
conventional pipelines, as shown in Fig. 1 (c).

In our proposed ResVR, HR ODIs are firstly embedded into LR images to
facilitate efficient transmission. Then, HR viewports are directly rendered
from LR ERP images on HMDs. This process ($\hat{\mathbf{I}}^{\text{LR-
ERP}}\mapsto\hat{\mathbf{I}}^{\text{HR-View}}$) does not need to produce HR
ERP images. To deal with the irregular correspondence between the ERP area and
the viewport, which hinders the joint optimization of downscaling and viewport
rendering using traditional image-level loss training methods, we develop a
discrete pixel sampling strategy. In each training iteration, this strategy is
used to randomly sample paired sets of ground truth pixels and the
reconstructed ones on viewports, thus making the end-to-end learning of our
entire ResVR pipeline feasible in implementation. Furthermore, to enhance our
ResVR’s awareness of the positions of different viewport pixels on the
spherical surface, we introduce a technique for spherical pixel shape
representation. This technique employs spherical differentiation to calculate
the geometric orientation and curvature of various viewport areas, offering
positional information that contains more precise spherical attributes than
existing 2D image representation methods (Lee et al., 2022). This advanced
representation effectively improves the quality of the final viewport,
especially in regions of high latitude and longitude. Extensive experiments on
various panoramic image datasets show that ResVR outperforms existing methods
in multiple viewport rendering tasks across different FoVs, view directions,
and resolutions. In summary, our contributions are:

❑ We propose ResVR, a novel framework for the comprehensive processing of
omnidirectional images, seamlessly integrates image Rescaling with Viewport
Rendering. Our ResVR effectively balances the transmission efficiency and
users’ visual experience.

❑ We develop a discrete pixel sampling strategy to tackle the complex
correspondence between viewport and equirectangular projection (ERP) areas
within our framework. This strategy makes the end-to-end training of our whole
processing pipeline feasible.

❑ We introduce a spherical pixel shape representation technique based on
spherical differentiation to guide viewport rendering, which significantly
enhances the visual quality of the final viewport.

❑ Extensive empirical evaluations on various panoramic image datasets exhibit
that ResVR consistently achieves new state-of-the-art visual quality while
maintaining a low transmission bitrate.

Figure 2. Overview of our proposed ResVR framework. The comprehensive ODI
processing of ResVR contains two sequential steps: (1) ODI Downscaling &
Compression and (2) Decompression & Viewport Rendering. (a) In the training
process, HR ERP patches $\mathbf{I}^{\text{HR-ERP-Patch}}$ are randomly
sampled through our proposed discrete pixel sampling strategy (Algo. 1) to
generate the guided LR patches $\mathbf{I}^{\text{LR-ERP-Patch}}$, query
coordinates $\mathbf{X}_{samp}$ and the set of ground truth pixels $S_{pix}$.
This strategy innovatively makes the end-to-end training of ResVR feasible in
implementation. (b) During inference, our trained ResVR model can be directly
applied for joined rescaling and viewpoint rendering of given HR ERP images
from the cloud server to user HMDs.

##  2\. Related Work

###  2.1. Omnidirectional Image Super-Resolution

Image super-resolution (SR) seeks to construct HR images from LR ones. Since
the advent of deep neural networks (DNNs) in SRCNN (Dong et al., 2015),
subsequent studies (Lim et al., 2017; Zhang et al., 2018b; Chen et al., 2023;
Liang et al., 2021a; Cheng et al., 2023; Zhang et al., 2018c; Wang et al.,
2021, 2023; Lin et al., 2023; Zhang et al., 2022; Yang et al., 2023a; Li et
al., 2022; Chen et al., 2024; Liu et al., 2021) have significantly advanced SR
performance beyond traditional methods. In the specific context of
omnidirectional image super-resolution (ODISR), DNN-based approaches have been
tailored to account for the unique latitude-based characteristics of ODIs (Liu
et al., 2020; Nishiyama et al., 2021; Ozcinar et al., 2019; Cao et al., 2023b,
c; Deng et al., 2021; Yoon et al., 2022; Yu et al., 2023; Sun et al., 2023;
Cao et al., 2023a; Ai et al., 2023; Li et al., 2024). For example, LAU-Net
(Deng et al., 2021) divides the entire ERP image into latitude-based patches
for separate upscaling. SphereSR (Yoon et al., 2022) introduces the spherical
local implicit image function (SLIIF) alongside a novel feature extraction
module to leverage information from arbitrary projection types. OSRT (Yu et
al., 2023) employs a distortion-aware transformer targeting dimension-related
distortions in ERP images. OPDN (Sun et al., 2023) introduces a dual-stage
framework incorporating a position-aware deformable network. Despite these
noteworthy advancements, the majority of these methodologies presuppose a
fixed downscaling approach (e.g. bicubic (Mitchell and Netravali, 1988)) and
overlook high-frequency components from HR inputs, thus limiting the quality
of reconstructed details in SR ODIs.

###  2.2. Image Rescaling

Different from SR, image rescaling focuses on downscaling HR images to create
visually pleasing LR images that retain essential information for accurate HR
reconstruction. Recently, invertible neural network (INN) (Behrmann et al.,
2019; Chen et al., 2019; Ouyang et al., 2022; Zhang et al., 2024) becomes a
representative framework for image rescaling (Cheng et al., 2021; Xiao et al.,
2020, 2023; Liang et al., 2021b; Yang et al., 2023b; Xu et al., 2023; Pan et
al., 2022; Sun and Chen, 2020), offering a direct route to inversely map the
downscaled images back to HR. For instance, IRN (Xiao et al., 2020, 2023) is
the first attempt to model image downscaling and upscaling using invertible
transmissions. Liang et al. (Liang et al., 2021b) formulate high-frequency
components in INNs as a conditional distribution on the LR image.
HyberThumbnail (Qi et al., 2023) employs an asymmetric encoder-decoder
architecture for real-time reconstruction of 6K images and also optimizes the
JPEG compression process (Wu et al., 2021; Wallace, 1991; Xing et al., 2024).
Very recently, DINN (Guo et al., 2023) makes the first attempt to apply image
rescaling to ODI and highlights the significance of leveraging ERP’s latitude
characteristics by developing a latitude-aware conditional mechanism. However,
existing ODI rescaling methods focus solely on improving the quality of ERP
images. In contrast, our ResVR innovatively optimizes the quality of the final
viewport, offering new improvements orthogonal to previous ODI rescaling
methodologies.

###  2.3. Viewport Rendering of ODIs

ODIs are designed to encapsulate a full spherical view, enabling an immersive
viewing experience. However, when viewed through HMDs towards a particular
direction, only a specific viewport is displayed (Jabar et al., 2017, 2019).
Achieving high resolution and quality in these viewports is crucial for
immersive experiences, as highlighted by various ODI visual quality assessment
techniques (Xu et al., 2020a; Yang et al., 2022a, b; Xu et al., 2020b). As a
widely adopted method for viewport rendering (Carroll et al., 2009),
perspective projection employs a series of pixel mapping and resampling
operations to effectively implement image warping. To reduce the
interpolation-induced blurriness and artifacts, SRWarp (Son and Lee, 2021)
reinterprets image warping as an SR problem and introduces a differentiable
warping module. LTEW (Lee et al., 2022) uses a continuous neural
representation (Chen et al., 2021; Lee and Jin, 2022; Kim et al., 2024) for
image warping by taking advantage of both Fourier features and spatially-
varying Jacobian matrices. LeRF (Li et al., 2023) assigns spatially varying
steerable resampling functions to pixels, learning their orientations for
continuous function prediction. Different from the above methods that focus on
warping, our ResVR considers comprehensive ODI processing, serving as a new
framework while enjoying high viewpoint rendering quality.

##  3\. Methodology

In this section, we begin with a concise review of the viewport rendering
process (Sec. 3.1). Following this, we provide an overview of our ResVR
framework (Sec. 3.2), which is illustrated in Fig. 2. We then elaborate on the
proposed discrete pixel sampling strategy (Sec. 3.3) and spherical pixel shape
representation technique (Sec. 3.4). The training objectives are detailed in
Sec. 3.5.

###  3.1. Preliminaries of Viewport Rendering

ODIs inherently provide a full spherical view. However, when viewed through
HMDs directed towards a specific direction, only the corresponding viewport is
displayed. This viewport appears as a 2D image, derived through perspective
projection (Larsen and Fejfar, 1974) from a segment of the spherical image. To
formalize this process, we consider the view direction in spherical
coordinates $(\theta_{c},\phi_{c})$, along with the horizontal and vertical
fields of view $(F_{h},F_{v})$, and the height and width of the viewport
$(h_{v},w_{v})$. An invertible coordinate mapping
$f:\mathbf{X}\mapsto\mathbf{Y}$ is established, where
$\mathbf{X}:=\\{\mathbf{x}|\mathbf{x}\in\mathbb{R}^{2}\\}$ denotes the
coordinate space of ERP, and
$\mathbf{Y}:=\\{\mathbf{y}|\mathbf{y}\in\mathbb{R}^{2}\\}$ represents the
coordinate space of the viewport to be rendered. In practice, the coordinates
$\mathbf{Y}_{view}$ on the viewport are initially determined by
$(h_{v},w_{v})$, and then the corresponding coordinates on ERP are obtained
through backward mapping $\mathbf{X}_{view}=f^{-1}(\mathbf{Y}_{view})$. The
rendering of the viewport is achieved through resampling techniques, such as
interpolation, ensuring that this process remains fully differentiable.
Additional mathematical details of perspective projection and viewport
rendering are elaborated in the supplementary material.

###  3.2. Overview of ResVR

An overview of proposed joint Rescaling and Viewport Rendering (ResVR) of ODIs
is presented in Fig. 2. The ODI processing of ResVR contains two steps: (1)
ODI Downscaling & Compression: An HR ERP image is firstly downsampled and
compressed to an LR ERP JPEG image for efficient transmission from cloud
server to user HMDs, and (2) Decompression & Viewport Rendering: The LR ERP
image is then decompressed and rendered to HR viewports on HMDs through our
viewport rendering (VR) module.

ODI Downscaling & Compression: Given an HR ERP image $\mathbf{I}^{\text{HR-
ERP}}\in\mathbb{R}^{3\times H\times W}$, its LR representation is firstly
generated through our downsampler, where $s$ is the rescaling factor. The
downsampler is a U-Net (Ronneberger et al., 2015) with dense blocks (Huang et
al., 2017). To further decrease the size of the transmitted image file, a JPEG
encoder is employed for the LR representation to obtain a JPEG image.
Concretely, we follow (Qi et al., 2023) to predict adaptive quantization
tables for each image. Finally, the LR JPEG image is obtained, and adaptive
quantization tables and quantized DCT coefficients are also encoded into the
JPEG file. More details about the downsampler, the adaptive quantization table
prediction module, and the training process of learned compression are
provided in the supplementary material.

Figure 3. Illustration of the VR module, which consists of an encoder
$\mathcal{E}$, a local texture estimator $h_{\psi}$, and an MLP decoder
$\mathcal{D}$. Given query coordinates $\mathbf{X}$, it directly predicts
$\hat{\mathbf{I}}^{\text{HR-View}}$ from $\hat{\mathbf{I}}^{\text{LR-ERP}}$
without the need to produce an HR ERP image. Figure 4. Training ResVR end-to-
end faces challenges due to the mismatch in shapes between the ERP image patch
($\mathbf{I}^{\text{HR-ERP-Patch}}$) and the viewport ($\mathbf{I}^{\text{HR-
View}}$). In Situation #1, although we obtain $\mathbf{I}^{\text{HR-View}}$
with a rectangular image shape, its corresponding $\mathbf{I}^{\text{LR-ERP-
Patch}}$ does not have a rectangular image shape, preventing its use in
supervising the downscaling process. Situation #2 experiences the opposite
issue. Both two situations are impractical for training. In contrast, our
method utilizes a novel discrete pixel sampling strategy (Algo. 1) to make
end-to-end training feasible.

Decompression & Viewport Rendering: After receiving the LR JPEG image,
$\hat{\mathbf{I}}^{\text{LR-
ERP}}\in\mathbb{R}^{3\times\frac{H}{s}\times\frac{W}{s}}$ is firstly
reconstructed by the JPEG decoder through inverse discrete cosine
transformation (IDCT). Then our goal is to render high-resolution viewport
$\hat{\mathbf{I}}^{\text{HR-View}}$ $\in\mathbb{R}^{3\times h_{v}\times
w_{v}}$ directly from $\hat{\mathbf{I}}^{\text{LR-ERP}}$. Inspired by recent
implicit neural representation methods (Chen et al., 2021; Lee et al., 2022;
Lee and Jin, 2022), we develop a viewport rendering (VR) module to predict
pixel values of the query coordinates $\mathbf{X}_{view}$ from the latent
space of input $\hat{\mathbf{I}}^{\text{LR-ERP}}$ instead of using traditional
interpolation methods. As depicted in Fig. 3, our VR module consists of an
encoder $\mathcal{E}$, a local texture estimator
$h_{\psi}=\\{h_{a},h_{f},h_{p}\\}$, and an MLP decoder $\mathcal{D}$.
$h_{\psi}$ is a learnable dominant-frequency estimator, which is capable of
characterizing image textures in 2D Fourier space (Lee and Jin, 2022). Here,
$h_{a}$ is an amplitude estimator ($\mathbb{R}^{C}\mapsto\mathbb{R}^{256}$),
$h_{f}$ is a frequency estimator ($\mathbb{R}^{C}\mapsto\mathbb{R}^{2\times
128}$), and $h_{p}$ is a phase estimator
($\mathbb{R}^{10}\mapsto\mathbb{R}^{128}$). Concretely, for a query point
$\mathbf{x}\in\mathbf{X}_{view}$, the estimating function $h_{\psi}$ is
defined as:

(1)
$h_{\psi}(\mathbf{z}_{j},\delta_{\mathbf{x}},\mathbf{s}(\mathbf{x}))=h_{a}(\mathbf{z}_{j})\otimes\left[\begin{aligned}
\cos\\{\pi(<h_{f}(\mathbf{z}_{j}),\delta_{\mathbf{x}}>+h_{p}(\mathbf{s}(\mathbf{x})))\\}\\\
\sin\\{\pi(<h_{f}(\mathbf{z}_{j}),\delta_{\mathbf{x}}>+h_{p}(\mathbf{s}(\mathbf{x})))\\}\end{aligned}\right],$

where $\mathbf{z}=\mathcal{E}(\hat{\mathbf{I}}^{\text{LR-ERP}})$. Denote $j$
as a pixel index of the $\hat{\mathbf{I}}^{\text{LR-ERP}}$, $\mathbf{x}_{j}$
and $\mathbf{z}_{j}$ are the corresponding coordinates and latent variable of
$\hat{\mathbf{I}}^{\text{LR-ERP}}$, respectively. $\delta_{\mathbf{x}}$ is a
local grid calculated by $\delta_{\mathbf{x}}=\mathbf{x}-\mathbf{x}_{j}$.
$<\cdot,\cdot>$ is an inner product, and $\otimes$ denotes element-wise
multiplication. To be noted, $\mathbf{s}(\mathbf{x})$ is the spherical pixel
shape representation of the query coordinate $\mathbf{x}$, which is important
for providing spatial-varying priors (Lee et al., 2022) for our VR module and
will be elaborated on in Sec. 3.4. Finally, our VR module predicts the RGB
values of a coordinate $\mathbf{y}=f(\mathbf{x})$ on the viewport as:

(2) $\hat{\mathbf{I}}^{\text{HR-
View}}[\mathbf{y}]=\mathcal{F}_{B}(\hat{\mathbf{I}}^{\text{LR-
ERP}})+\sum_{j\in\mathcal{J}}\omega_{j}\mathcal{D}(h_{\psi}(\mathbf{z}_{j},\delta_{\mathbf{x}},\mathbf{s}(\mathbf{x}))),$

where $\mathcal{F}_{B}$ is a bilinear interpolation operator to stabilize the
convergence of network training and aid the VR module in learning high-
frequency details. $\mathcal{J}$ is a neighborhood set of $\mathbf{x}$,
defined as
$\mathcal{J}=\\{j|j=\mathbf{x}+[\frac{ms}{W},\frac{ns}{H}],m,n\in\\{-1,1\\}\\}$,
while $\omega_{j}$ is a local ensemble coefficient. More details of our VR
module are provided in the supplementary material. Different from existing ODI
rescaling methods which first produce $\hat{\mathbf{I}}^{\text{HR-ERP}}$ and
then obtain HR viewports through traditional interpolation methods, our ResVR
directly renders the final HR viewports from $\hat{\mathbf{I}}^{\text{LR-
ERP}}$ through our VR module.

Algorithm 1 Discrete pixel sampling strategy
$\mathbf{I}^{\text{HR-ERP}}$, $s$: HR ERP image and the downscaling factor
$(a,b)$, $p$: left top coordinate and the size of cropped patch
$(\theta_{c},\phi_{c})$, $(F_{h},F_{v})$: view direction and FoVs of the
viewport
$(h_{v},w_{v})$, $\mathbf{Y}_{view}$: shape and the coordinate space of the
viewport

function DisSamp($a$, $b$, $p$, $\theta_{c}$, $\phi_{c}$, $F_{h}$, $F_{v}$,
$h_{v}$, $w_{v}$, $\mathbf{Y}_{view}$, $\mathbf{I}^{\text{HR-ERP}}$)

$\mathbf{I}^{\text{HR-ERP-Patch}}$ $\leftarrow$
CropPatch($\mathbf{I}^{\text{HR-ERP}}$, $a$, $b$, $p$)

$f$ $\leftarrow$ GetTransform($\theta_{c}$, $\phi_{c}$, $F_{h}$, $F_{v}$,
$h_{v}$, $w_{v}$)

$\mathbf{X}_{view}$ $\leftarrow$ $f^{-1}(\mathbf{Y}_{view})$$\triangleright$
Inverse mapping from viewport to ERP

$\mathbf{X}_{view}^{\prime}$ $\leftarrow$
FilterWithBounds($\mathbf{X}_{view}$, $a$, $b$, $p$) $\triangleright$ Eq. (3)

if $|\mathbf{X}_{view}^{\prime}|>N$ then

$\mathbf{X}_{view}^{\prime}$ $\leftarrow$
RandomSample($\mathbf{X}_{view}^{\prime}$, $N$)

end if

$\mathbf{X}_{samp}$ $\leftarrow$
CoordSpaceTransform($\mathbf{X}_{view}^{\prime}$, $a$, $b$, $p$)
$\triangleright$ Eq. (4)

$S_{pix}$ $\leftarrow$ BicubicSample($\mathbf{I}^{\text{HR-ERP-Patch}}$,
$\mathbf{X}_{samp}$)

$\mathbf{I}^{\text{LR-ERP-Patch}}$ $\leftarrow$
BicubicDownscale($\mathbf{I}^{\text{HR-ERP-Patch}}$, s) return
$\mathbf{X}_{samp}$, $S_{pix}$, $\mathbf{I}^{\text{LR-ERP-Patch}}$

end function

###  3.3. Discrete Pixel Sampling Strategy

Motivation. In image rescaling and SR tasks, the ground truth HR images
$\mathbf{I}^{\text{HR-ERP}}$ are typically cropped into patches
$\mathbf{I}^{\text{HR-ERP-Patch}}$ to reduce GPU memory overhead while
increasing the diversity of training data. The goals of our ResVR’s end-to-end
training process include: (1) optimizing the downscaling process
($\mathbf{I}^{\text{HR-ERP-Patch}}$ $\mapsto$ $\hat{\mathbf{I}}^{\text{LR-ERP-
Patch}}$), and (2) optimizing the rendering process
($\hat{\mathbf{I}}^{\text{LR-ERP-Patch}}$ $\mapsto$
$\hat{\mathbf{I}}^{\text{HR-View}}$). In this framework, the former process
requires the corresponding $\mathbf{I}^{\text{LR-ERP-Patch}}$ as supervision,
while the latter process needs the corresponding $\mathbf{I}^{\text{HR-View}}$
as supervision. However, due to the natural geometric properties of ODIs’
different projection types, there exists a challenge of shape mismatch between
a viewport and its corresponding ERP area. As depicted in Situations #1 and #2
in Fig. 4, one can not obtain $\mathbf{I}^{\text{LR-ERP-Patch}}$ and
$\mathbf{I}^{\text{HR-View}}$ that are both with rectangular image shape at
the same time. As a result, conventional image-level loss can not be directly
used for end-to-end training of ResVR. To address this, we innovatively
propose a discrete pixel sampling strategy (DPS), as shown in “Ours” of Fig.
4.

Method. Denoting $p$ as both the height and width of training patches, our
core idea is to keep the LR ERP patch with a rectangular image shape
($\mathbf{I}^{\text{LR-ERP-
Patch}}\in\mathbb{R}^{3\times\frac{p}{s}\times\frac{p}{s}}$ ), while
discretely sampling $N$ pixels $S_{pix}\in\mathbb{R}^{3\times N}$ with their
coordinates $\mathbf{X}_{samp}\in\mathbb{R}^{2\times N}$ in the irregular
area. Thanks to the continuous representation ability of implicit neural
representation methods (Chen et al., 2021), our VR module is able to predict
corresponding pixel values $\hat{S}_{pix}$ given coordinates
$\mathbf{X}_{samp}$. Hence, we use the sampled $S_{pix}$ as supervision for
the predicted $\hat{S}_{pix}$. An overview of our strategy is presented in
Algo. 1.

Specifically, given the left top coordinate $(a,b)$ of the training patch, the
view direction $(\theta_{c},\phi_{c})$, FoVs ($F_{h}$ and $F_{v}$), shapes
($h_{v}$, $w_{v}$), and query coordinates ($\mathbf{Y}_{view}$) of the desired
viewport, we first determine the coordinate mapping $f$. Then the
corresponding coordinates $\mathbf{X}_{view}$ in the ERP space can be obtained
through inverse coordinate mapping as
$\mathbf{X}_{view}=f^{-1}(\mathbf{Y}_{view})$. By setting appropriate view
direction and FoVs, it is possible to ensure that some elements in
$\mathbf{X}_{view}$ lie in the area of the cropped ERP patch. Recall that $p$
is the size of the cropped training patch, $\mathbf{X}_{view}$ is then
filtered to preserve the subset overlapped with the patch as follows:

(3)
$\mathbf{X}_{view}^{\prime}=\\{\mathbf{x}\in\mathbf{X}_{view}|a\leq\mathbf{x}_{1}<(a+p),b\leq\mathbf{x}_{2}<(b+p)\\},$

where $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ represent the coordinates of a
point $\mathbf{x}$ in the ERP space. This filter operation ensures that every
point in $\mathbf{X}_{view}^{\prime}$ can find its corresponding latent
variable extracted from $\hat{\mathbf{I}}^{\text{LR-ERP-Patch}}$. To balance
the number of pixels sampled from different FoVs and view directions, if the
filtered $\mathbf{X}_{view}^{\prime}$ contains more than $N$ pixel
coordinates, only $N$ elements will be randomly retained. Finally, to uniform
the coordinate correspondence between LR and HR patches,
$\mathbf{X}_{view}^{\prime}\subseteq[0,H)\times[0,W)$ is converted from the
whole HR ERP coordinate space to the patch coordinate space, and normalized to
$\mathbf{X}_{samp}\subseteq[-1,1)\times[-1,1)$, as:

(4) $\displaystyle\mathbf{X}_{samp}$
$\displaystyle=\\{T(\mathbf{x})|\mathbf{x}\in\mathbf{X}_{view}^{\prime}\\},$
$\displaystyle\text{where}\ \ T(\mathbf{x})$
$\displaystyle=2\left(\left(\left(\mathbf{x}_{1},\mathbf{x}_{2}\right)-(a,b)\right)/p\right)-1.$

As a result, $S_{pix}$ are sampled through bicubic interpolation as
$S_{pix}=\mathcal{F}_{bicubic}(\mathbf{I}^{\text{HR-ERP-
Patch}},\mathbf{X}_{samp})$. By adopting Algo. 1, $\mathbf{I}^{\text{LR-ERP-
Patch}}\in\mathbb{R}^{3\times\frac{p}{s}\times\frac{p}{s}}$ can be used as
supervision of the predicted $\hat{\mathbf{I}}^{\text{LR-ERP-Patch}}$ , and
$S_{pix}\in\mathbb{R}^{3\times N}$ can be used as supervision of the predicted
$\hat{S}_{pix}$.

Figure 5. Illustration of our proposed spherical pixel shape representation
(SSR) technique. We illustrate using a point $\mathbf{y}$ on the viewport. The
inverse mapping is firstly applied for $\mathbf{y}$ and its eight nearest
neighbors to get $\mathbf{x}$ and its neighbors on ERP. Then these points are
transformed into sphere coordinates
$\\{\mathbf{p}_{1},\mathbf{p}_{2},\cdots,\mathbf{p}_{9}\\}$, which are used
for calculating numerical derivatives to estimate the pixel shape
representation $\mathbf{s}(\mathbf{x})$, according to proposed spherical
central difference method in Eqs. (6) and (7).

###  3.4. Spherical Pixel Shape Representation

Motivation. Pixel shape representations described by grid orientation and
curvature provide informative geometric spatial-varying priors for image
warping tasks (Lee et al., 2022). Existing shape representation methods are
designed for 2D images. However, different from 2D images, ERP is an unfolding
of the spherical surface along meridians, which leads to the fact that the
adjacent pixels on the sphere can be far apart in the ERP image. As a result,
the previous 2D shape representation methods fall in the high-
latitude/-longitude areas due to the nature properties of (1) latitude-related
distortion and (2) wraparound consistency of ERP images (Fig. 7 in Sec. 4.3).
Therefore, we develop a spherical pixel shape representation (SSR) technique
to solve this challenge. As Fig. 5 shows, SSR leverages the information of
transformed coordinates on the sphere as a more effective shape representation
to guide the viewport rendering process.

Method. In image warping tasks, the first-order partial derivatives (i.e.
Jacobian matrix) and the second-order partial derivatives (i.e. Hessian
matrix), describe the orientation and curvature of pixels resulting from the
transformation, respectively (Lee et al., 2022). These two matrices provide
informative geometric spatial-varying priors to guide the process of viewport
rendering. Inspired by this, for a point $\mathbf{x}$ on the original ERP
plane, we represent its pixel shape $\mathbf{s}(\mathbf{x})$ with the gradient
of the inverse coordinate transformation $f^{-1}$. Specifically, we start with
the point $\mathbf{y}=f(\mathbf{x})$ on the viewport plane, and the
corresponding Jacobian matrix $\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})$
and Hessian matrix $\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$ are
analytically computed as:

(5) $\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})=\left[\begin{aligned}
\frac{\partial\mathbf{x}_{1}}{\partial u}\
\frac{\partial\mathbf{x}_{1}}{\partial v}\\\
\frac{\partial\mathbf{x}_{2}}{\partial u}\
\frac{\partial\mathbf{x}_{2}}{\partial
v}\end{aligned}\right],\quad\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})=\left[\begin{aligned}
\frac{\partial^{2}\mathbf{x}_{1}}{\partial^{2}u}\
\frac{\partial^{2}\mathbf{x}_{1}}{\partial u\partial v}\\\
\frac{\partial^{2}\mathbf{x}_{2}}{\partial v\partial u}\
\frac{\partial^{2}\mathbf{x}_{2}}{\partial^{2}v}\end{aligned}\right],$

where $\mathbf{x}=f^{-1}(\mathbf{y})=(\mathbf{x}_{1},\mathbf{x}_{2})$
represents the coordinates in the original ERP plane. We propose a spherical
pixel shape representation (SSR) technique to numerically estimate
$\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})$ and
$\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$ based on spherical
differentiation. Specifically, as depicted in Fig. 5, the inverse coordinate
transformation $f^{-1}$ is firstly applied to $\mathbf{y}$ and its eight
nearest points $(\mathbf{y}+[\frac{m}{w_{v}},\frac{n}{h_{v}}]$ with
$m,n\in\\{-1,0,1\\})$ to get $\mathbf{x}$ and its neighborhood on the ERP
plane. Different from existing 2D shape representation methods (Lee et al.,
2022) which estimate the shape of $\mathbf{x}$ directly on the ERP image
plane, we further transform $\mathbf{x}$ and its neighborhood to the sphere
denoted by the set
$\\{\mathbf{p}_{i}|\mathbf{p}_{i}=(\theta_{i},\phi_{i}),i=1,2,\ldots,9\\}$,
and innovatively calculate numerical derivatives to estimate
$\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})$ and
$\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$ by the proposed spherical
central difference method, overriding Eq. (5) as:

(6) $\displaystyle\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})$
$\displaystyle\approx\left[\begin{aligned} D(\mathbf{p}_{6},\mathbf{p}_{4})\\\
D(\mathbf{p}_{2},\mathbf{p}_{8})\end{aligned}\right],$
$\displaystyle\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$
$\displaystyle\approx\left[\begin{aligned}
D(\mathbf{p}_{6},\mathbf{p}_{5})+D(\mathbf{p}_{5},\mathbf{p}_{4})\ \
D(\mathbf{p}_{3},\mathbf{p}_{1})+D(\mathbf{p}_{9},\mathbf{p}_{7})\\\
D(\mathbf{p}_{3},\mathbf{p}_{1})+D(\mathbf{p}_{9},\mathbf{p}_{7})\ \
D(\mathbf{p}_{2},\mathbf{p}_{5})+D(\mathbf{p}_{5},\mathbf{p}_{8})\end{aligned}\right],$

where $D(\mathbf{p}_{i},\mathbf{p}_{j})$ calculates the distance between
$\mathbf{p}_{i}$ and $\mathbf{p}_{j}$ in the spherical coordinate system as:

(7) $\displaystyle D(\mathbf{p}_{i},\mathbf{p}_{j})=\left(\begin{aligned}
\min(|\phi_{j}-\phi_{i}|,&\ 2\pi-|\phi_{j}-\phi_{i}|)\\\
\min(|\theta_{j}-\theta_{i}|,\
2\pi-|&\theta_{j}-\theta_{i}|)\cdot\cos\left(\left(\theta_{i}+\theta_{j}\right)/2\right)\end{aligned}\right)^{\top},$

where $|\cdot|$ represents the absolute value operation. The $\min(\cdot)$
operator ensures the computation of minor arc distances, and the average
latitude is utilized to adjust the longitudinal distance to account for the
convergence of meridians. We follow (Lee et al., 2022) to use six elements in
$\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$, and finally
$\mathbf{s}(\mathbf{x})\in\mathbb{R}^{10}$ is obtained by concatenating and
flattening $\widetilde{\mathbf{J}}_{f^{-1}}(\mathbf{y})$ and
$\widetilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$. As a result,
$\mathbf{s}(\mathbf{x})$ serves as an auxiliary input, together with the LR
latent representation $\mathbf{z}=\mathcal{E}(\hat{\mathbf{I}}^{\text{LR-
ERP}})$ to predict final viewports through the VR module.

Table 1. Quantitative evaluation of rendered viewports with
$(F_{h},F_{v})=(120^{\circ},90^{\circ})$ and $(w_{v},h_{v})=(2048,1536)$. We
keep bpp around 0.3 on different datasets. The WS-PSNR is evaluated for
methods that need to explicitly get HR ERP to render the viewport. Focusing
solely on the quality of ERP images results in the sub-optimal visual
experience of final viewports. Throughout this paper, the best and second-best
results of each test setting are highlighted in bold red and underlined blue,
respectively.

Method | ODI-SR (Deng et al., 2021) | SUN 360 (Xiao et al., 2012)
---|---|---
Down & Compression & Up & Render | bpp$\downarrow$ | WS-PSNR | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | bpp$\downarrow$ | WS-PSNR | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
Bicubic & JPEG & Bicubic | 0.29 | N/A | 27.98 | 0.7897 | 0.4815 | 0.28 | N/A | 28.06 | 0.8077 | 0.4642
Bicubic & JPEG & OSRT (Yu et al., 2023) & Bicubic | 0.29 | 25.73 | 28.42 | 0.7975 | 0.4395 | 0.28 | 26.09 | 28.84 | 0.8202 | 0.5510
HyperThumbnail (Qi et al., 2023) & Nearest | 0.30 | 27.84 | 30.13 | 0.8314 | 0.4197 | 0.29 | 28.97 | 31.16 | 0.8601 | 0.3768
HyperThumbnail (Qi et al., 2023) & Bilinear | 0.30 | 27.84 | 30.82 | 0.8475 | 0.3397 | 0.29 | 28.97 | 32.20 | 0.8775 | 0.2792
HyperThumbnail (Qi et al., 2023) & Bicubic | 0.30 | 27.84 | 31.00 | 0.8515 | 0.3222 | 0.29 | 28.97 | 32.54 | 0.8822 | 0.2610
ResVR (Ours) | 0.30 | N/A | 31.39 | 0.8568 | 0.3026 | 0.29 | N/A | 32.95 | 0.8862 | 0.2462

Ground Truth | Bic & JPEG & Bic | Bic & JPEG & OSRT & Bic | HyperThumbnail & Bic | ResVR (Ours)
---|---|---|---|---
|  |  |  |
PSNR $\uparrow$ / LPIPS $\downarrow$ | 24.66/0.4771 | 25.45/0.4005 |  29.24/0.2427 |  30.08/0.2211
|  |  |  |
PSNR $\uparrow$ / LPIPS $\downarrow$ | 25.79/0.5191 | 27.12/0.3910 |  30.98/0.2148 |  31.90/0.1994
Figure 6. Comparisons of two rendered viewports from ODI-SR (Deng et al.,
2021) (“img_005”, top) and SUN 360 (Xiao et al., 2012) (“img_046”, bottom),
with $(\theta=0^{\circ},\phi=90^{\circ})$ and
$(\theta=45^{\circ},\phi=180^{\circ})$, respectively. The viewports are with
FoVs $(F_{h},F_{v})=(120^{\circ},90^{\circ})$ and resolutions
$(w_{v},h_{v})=(2048,1536)$. “Bic” stands for Bicubic interpolation. Please
zoom in for more details.

###  3.5. Training Objectives

Thanks to the discrete pixel sampling strategy (Sec. 3.3) and the fully
differentiable pipeline (Fig. 2), the processes of downscaling, compression,
and viewport rendering can be jointly optimized end-to-end, which aligns the
learning of network parameters with our goals of obtaining high-quality
viewport while reducing the transmission overhead. The total loss is a
weighted sum of a pixel-level reconstruction loss, an LR guidance loss, and a
bitrate loss as:

(8)
$\mathcal{L}=\mathcal{L}_{\text{pix}}+\lambda_{1}\mathcal{L}_{\text{guide}}+\lambda_{2}\mathcal{L}_{\text{bpp}},$

where $\lambda_{1}$ and $\lambda_{2}$ are two trade-off parameters.

Pixel-level reconstruction and guidance loss. In the training phase, we employ
Algo. 1 to get $S_{pix}$ as the ground truth and use the VR module to predict
the corresponding values $\hat{S}_{pix}$ given $X_{samp}$.

(9) $\mathcal{L}_{\text{pix}}=\frac{||\hat{S}_{pix}-S_{pix}||_{1}}{N},$

where $N$ is the number of sampled pixels. Additionally, following (Xiao et
al., 2020, 2023; Qi et al., 2023), an $L_{2}$ guidance loss on the LR ERP
patch is defined as:

(10) $\mathcal{L}_{guide}=\frac{||\hat{\mathbf{I}}^{\text{LR-ERP-
Patch}}-\mathbf{I}^{\text{LR-ERP-Patch}}||^{2}_{2}}{(p/s)\times(p/s)}.$

Bitrate loss. To optimize the size of the transmitted JPEG image, we firstly
follow (Ballé et al., 2016) to estimate the rate $R$ of the quantized
coefficients $\widetilde{C}$ with differentiable fully factorized entropy
models as: $R=\mathbb{E}_{x\sim p_{x}}[-\log p_{L}(\widetilde{C}_{y})-\log
p_{C}(\widetilde{C}_{Cb})-\log p_{C}(\widetilde{C}_{Cr})]$, where $p_{L}$ and
$p_{C}$ are two fully factorized entropy models for luma and chroma
coefficient maps, respectively. Then the bpp of the transmitted LR ERP JPEG
image is calculated as:

(11) $\mathcal{L}_{bpp}=\frac{R}{H\times W}.$

##  4\. Experiments

###  4.1. Experimental Setup

Implementation details. We follow the previous work (Qi et al., 2023) to put
more computation into the downsampler to keep the VR module lightweight. The
VR module is composed of a lightweight feature extractor and a tiny MLP.
Please refer to our supplementary materials for more details of our network
architecture.

Training details. The downscaling factor $s$ is fixed to 4, and patch size $p$
of random cropped $\mathbf{I}^{\text{HR-ERP-Patch}}$ is set to 256. To ensure
the overlapping of $\mathbf{I}^{\text{HR-ERP-Patch}}$ and discretely sampled
points in Algo. 1, we calculate the viewport center $(\theta_{c},\phi_{c})$
according to the center of cropped patch. To enable ResVR to handle different
resolutions and FoVs, we randomly sample FoVs and resolutions during training.
Specifically, the FoVs $(F_{h},F_{v})$ are randomly sampled from
$\\{80^{\circ},90^{\circ},100^{\circ},110^{\circ},120^{\circ}\\}$ and the
resolutions of viewport $(h_{v},w_{v})$ are randomly sampled from
$\\{512,576,640,768,832,960,1024\\}$. The number of sampled pixels $N$ is set
to $25600$, and the batch size is set to $16$. All experiments are conducted
on one V100 GPU. The network is trained for $5\times 10^{5}$ iterations with
learning rate $2\times 10^{-4}$. $\lambda_{1}$ and $\lambda_{2}$ are set to
$0.6$ and $0.01$ for all experiments, respectively.

Datasets and evaluation metrics. ODI-SR dataset (Deng et al., 2021) and SUN360
Panorama dataset (Xiao et al., 2012) are used in our experiment. We follow the
data split setting in (Deng et al., 2021) and train on the ODI-SR training
set. For evaluation of transmission efficiency, we follow (Qi et al., 2023) to
use the real file size of JPEG for evaluating the bitrate:
$\text{bpp}=\mathbb{E}_{x\sim p_{x}}[\text{filesize}/(H\times W)]$. For
evaluation of the quality of rendered viewports, we choose ten different view
directions, get the ground truth image using bicubic interpolation, and
calculate the average PSNR, SSIM, and LPIPS (Zhang et al., 2018a). More
details are provided in the supplementary materials. For those competing
methods that need to explicitly produce HR ERP images to render final
viewports, we evaluate the WS-PSNR (Sun et al., 2017) of their predicted
$\hat{\mathbf{I}}^{\text{HR-ERP}}$.

###  4.2. Comparison with State-of-the-Art Methods

We compare three categories of methods: (1) a baseline method which downscales
with Bicubic interpolation, compresses with standard JPEG codec, and renders
viewports with Bicubic interpolation; (2) SR pipeline which downscales with
Bicubic, compresses with standard JPEG codec, upscales with state-of-the-art
methods (Yu et al., 2023), and uses bicubic interpolation for viewport
rendering; (3) Rescaling pipeline which firstly uses state-of-the-art
asymmetric rescaling methods (Qi et al., 2023) and renders viewports with
Bicubic interpolation. For a fair comparison, we retrain SR methods (Yu et
al., 2023) and rescaling methods (Qi et al., 2023) with our training sets and
constrain their bpp around 0.3 by adjusting the quality factor of JPEG
compression in all baselines.

Tab. 1 presents the quantitative comparisons of the quality of rendered
viewports with $(F_{h},F_{v})=(120^{\circ},90^{\circ})$ and
$(w_{v},h_{v})=(2048,1536)$ among different methods. Taking advantage of
directly optimizing the final viewport through end-to-end training, ResVR
outperforms previous methods by reconstruction accuracy (about 0.4dB gain on
PSNR) and with better realness (about 0.02 gain on LPIPS). Notably, even
though existing rescaling methods can achieve PSNR of about 28dB, they only
obtain sub-optimal results due to the lack of awareness and optimization of
the viewport rendering process, which demonstrates the fact that focusing
solely on the quality of ERP images results in sub-optimal visual experience
of viewports. ResVR directly optimizes the rendered viewports without the need
to predict HR ERP and achieves SOTA performance. Fig. 6 further provides the
visual comparison of rendered viewports. It can be seen that our ResVR
reconstructs enhanced details with fewer artifacts (e.g. the text “labatt’s”
in the top image and the lines in the bottom image) compared to other methods.
Additionally, ResVR shows better wraparound continuity at the location of the
seams (e.g., pipe in the zoomed-in area) in the bottom case. We attribute this
to the awareness of arbitrary view directions during training and our MLP’s
continuous representation of images.

###  4.3. Ablation Study

In this section, we analyze the effect of the proposed discrete pixel sampling
strategy and spherical pixel shape representation. We conduct experiments on
three ResVR variants: (1) Case $\\#1$: We train the rescaling process and VR
module separately with 2D shape representation; (2) Case $\\#2$: We train the
whole pipeline in an end-to-end manner by employing discrete pixel sampling
strategy with 2D shape representation; (3) The complete ResVR with our sphere
pixel shape representation. Quantitative results are shown in Tab. 2.

Effect of discrete pixel sampling strategy. Comparison $\\#1$ vs. $\\#2$
exhibits the effectiveness of the proposed discrete pixel sampling strategy
which enables end-to-end training of the whole pipeline, thus providing
substantial PSNR improvement of 4.12-4.44dB. To further analyze the sampled
coordinates and pixels, we visualize the sampled areas and pixels in the
supplementary materials.

Table 2. Ablation study on proposed discrete pixel sampling strategy (DPS) and
spherical pixel shape representation (SSR) with setting
$(F_{h},F_{v})=(90^{\circ},90^{\circ})$ and $(w_{v},h_{v})=(1024,1024)$.

Case | Test Set | DPS | SSR | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---|---|---
$\\#$1 |  | ✘ | ✘ | 27.35 | 0.8296 | 0.3321
$\\#$2 |  | ✔ | ✘ | 31.47 | 0.8560 | 0.2820
ResVR | ODISR | ✔ | ✔ | 31.87 | 0.8606 | 0.2696
$\\#$1 |  | ✘ | ✘ | 28.09 | 0.8503 | 0.3040
$\\#$2 |  | ✔ | ✘ | 32.53 | 0.8742 | 0.2443
ResVR | SUN360 | ✔ | ✔ | 33.08 | 0.8793 | 0.2324

Ground Truth |  Case $\\#$2 (w/o SSR) | ResVR (Ours)
---|---|---
|  |
$\theta=0^{\circ},\phi=0^{\circ}$ | 38.70/0.1892 | 38.97/0.1865
|  |
$\theta=-90^{\circ},\phi=0^{\circ}$ | 34.48/0.3103 | 39.19/0.2554
|  |
$\theta=0^{\circ},\phi=180^{\circ}$ | 36.89/0.2188 | 37.79/0.2037

Figure 7. Visual comparison of two variants (Case $\\#2$ and Ours). 2D shape
representation falls in high-latitude and high-longitude areas (highlighted by
red arrow). In contrast, our spherical representation ensures stable rendering
results in various directions with better PSNR (dB)$\uparrow$ and LPIPS
$\downarrow$.

Effect of spherical pixel shape representation. Comparison $\\#2$ vs. ResVR
reveals that the spherical shape representation improves PSNR by 0.40-0.55dB.
We observe that conventional 2D shape representation methods (Lee et al.,
2022) fall in high-longitude and high-latitude areas, as depicted in Fig. 7.
We attribute this to the following reasons: adjacent pixels on the viewport
should be very close to each other on the sphere. However, (1) for high-
latitude areas, due to the natural distortion of ERP, the distance between
corresponding points is very close to the original sphere but is very far on
the ERP image. (2) For high-longitude areas, due to the wraparound consistency
of the left and right ends of ERP, the corresponding points are originally
adjacent pixels on the sphere but are at both ends of the ERP image. The
traditional shape representation method based on 2D ERP ignores the spherical
characteristics, so in the above situations, it provides wrong pixel shape
priors for the VR module, thus leading to abnormal visual results. The
proposed SSR is performed on the native spherical surface, thus ensuring
stable and high-quality rendering quality in various view directions.

Table 3. Quantitative evaluation on viewports of different resolutions with
$(F_{h},F_{v})=(120^{\circ},120^{\circ})$. ResVR outperforms existing methods
under various resolutions. Bic: Bicubic interpolation, Res: Resolution, HT:
HyperThumbnail (Qi et al., 2023).

|  | ODI-SR (Deng et al., 2021) | SUN 360 (Xiao et al., 2012)
---|---|---|---
Method | Res | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
HT (Qi et al., 2023) & Bic |  | 30.89 | 0.8704 | 0.1976 | 32.08 | 0.8880 | 0.1630
ResVR (Ours) | 5122 | 31.03 | 0.8754 | 0.1888 | 32.16 | 0.8914 | 0.1527
HT (Qi et al., 2023) & Bic |  | 30.92 | 0.8498 | 0.2802 | 32.09 | 0.8718 | 0.2343
ResVR (Ours) | 10242 | 31.25 | 0.8579 | 0.2632 | 32.39 | 0.8782 | 0.2176
HT (Qi et al., 2023) & Bic |  | 30.93 | 0.8541 | 0.3230 | 32.08 | 0.8769 | 0.2713
ResVR (Ours) | 20482 | 31.32 | 0.8596 | 0.3046 | 32.48 | 0.8812 | 0.2567

###  4.4. Analysis

LR JPEG image. The file size and visual quality of LR JPEG images are also
important for efficient transmission and user preview. ResVR effectively
balances transmission efficiency with the final viewer experience through our
end-to-end training. Visualizations of LR images are provided in the
supplementary materials.

Arbitrary-resolution viewport rendering. Due to the continuous representation
ability (Chen et al., 2021; Lee et al., 2022; Lee and Jin, 2022) of the MLP in
our VR module, ResVR is able to render viewports of different resolutions
using only one set of parameters. Tab. 3 compares ResVR with SOTA methods on
different resolution settings. It can be seen that ResVR achieves better
performance at different resolutions.

##  5\. Conclusion

The rise of virtual reality and augmented reality applications has popularized
ODI rescaling to shrink the file size of images while maintaining their
quality. However, existing methods focus on improving the ERP image quality,
ignoring that HMDs use rendered viewports for display, not ERP images. This
focus on ERP quality alone will lead to compromised user experiences. To
address this, in this paper, we propose ResVR, a novel framework for joint
rescaling and viewport rendering of ODIs. In ResVR, we develop a discrete
pixel sampling strategy to tackle the irregular correspondence between the ERP
area and the viewport, enabling end-to-ending training of the whole pipeline.
A spherical pixel shape representation technique is introduced to serve as an
effective guidance for the rendering process, further enhancing the visual
quality of viewports. Experiments demonstrate that our ResVR achieves state-
of-the-art performance across different settings of FoVs, view directions, and
resolutions while keeping a low transmission overhead.

Limitations and future work. Existing ODI datasets are of relatively low
resolution and with compression artifacts, which limits the further
performance improvements of ResVR. Besides, although ResVR provides a
promising pipeline for comprehensive ODI processing and achieves SOTA
performance, it is crucial to extend ResVR to 360∘ video tasks (Wang et al.,
2024). We leave this for future work.

## References

  * (1)
  * Ai et al. (2023) Hao Ai, Zidong Cao, Yan-Pei Cao, Ying Shan, and Lin Wang. 2023.  HRDFuse: Monocular 360deg depth estimation by collaboratively learning holistic-with-regional depth distributions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13273–13282.
  * Ai et al. (2022) Hao Ai, Zidong Cao, Jinjing Zhu, Haotian Bai, Yucheng Chen, and Ling Wang. 2022.  Deep Learning for Omnidirectional Vision: A Survey and New Perspectives.  _arXiv preprint arXiv:2205.10468_ (2022).
  * Ballé et al. (2016) Johannes Ballé, Valero Laparra, and Eero P Simoncelli. 2016.  End-to-end optimized image compression.  _arXiv preprint arXiv:1611.01704_ (2016).
  * Behrmann et al. (2019) Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. 2019.  Invertible residual networks. In _Proceedings of the International Conference on Machine Learning (ICML)_. PMLR, 573–582.
  * Cao et al. (2023c) Mingdeng Cao, Chong Mou, Fanghua Yu, Xintao Wang, Yinqiang Zheng, Jian Zhang, Chao Dong, Gen Li, Ying Shan, Radu Timofte, et al. 2023c.  Ntire 2023 challenge on 360deg omnidirectional image and video super-resolution: Datasets, methods and results. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1731–1745.
  * Cao et al. (2023b) Zidong Cao, Hao Ai, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Lin Wang. 2023b.  OmniZoomer: Learning to Move and Zoom in on Sphere at High-Resolution. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 12897–12907.
  * Cao et al. (2023a) Zidong Cao, Hao Ai, and Lin Wang. 2023a.  360deg High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer.  _arXiv preprint arXiv:2304.07967_ (2023).
  * Carroll et al. (2009) Robert Carroll, Maneesh Agrawala, and Aseem Agarwala. 2009.  Optimizing content-preserving projections for wide-angle images.  _ACM Trans. Graph._ 28, 3 (2009), 43.
  * Chen et al. (2024) Bin Chen, Zhenyu Zhang, Weiqi Li, Chen Zhao, Jiwen Yu, Shijie Zhao, Jie Chen, and Jian Zhang. 2024.  Invertible Diffusion Models for Compressed Sensing.  _arXiv preprint arXiv:2403.17006_ (2024).
  * Chen et al. (2019) Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. 2019.  Residual flows for invertible generative modeling. In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Chen et al. (2023) X Chen, X Wang, J Zhou, and C Dong. 2023.  Activating More Pixels in Image Super-Resolution Transformer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22367–22377.
  * Chen et al. (2021) Yinbo Chen, Sifei Liu, and Xiaolong Wang. 2021.  Learning continuous image representation with local implicit image function. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 8628–8638.
  * Cheng et al. (2021) Ka Leong Cheng, Yueqi Xie, and Qifeng Chen. 2021.  Iicnet: A generic framework for reversible image conversion. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 1991–2000.
  * Cheng et al. (2023) Ming Cheng, Haoyu Ma, Qiufang Ma, Xiaopeng Sun, Weiqi Li, Zhenyu Zhang, Xuhan Sheng, Shijie Zhao, Junlin Li, and Li Zhang. 2023.  Hybrid transformer and cnn attention network for stereo image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW)_. 1702–1711.
  * Deng et al. (2021) Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, and Li Yang. 2021.  Lau-net: Latitude adaptive upscaling network for omnidirectional image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 9189–9198.
  * Dong et al. (2015) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015.  Image super-resolution using deep convolutional networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ 38, 2 (2015), 295–307.
  * Guo et al. (2023) Yichen Guo, Mai Xu, Lai Jiang, Leonid Sigal, and Yunjin Chen. 2023.  DINN360: Deformable Invertible Neural Network for Latitude-Aware 360deg Image Rescaling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 21519–21528.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017.  Densely connected convolutional networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4700–4708.
  * Jabar et al. (2017) Falah Jabar, João Ascenso, and Maria Paula Queluz. 2017.  Perceptual analysis of perspective projection for viewport rendering in 360° images. In _2017 IEEE International Symposium on Multimedia (ISM)_. IEEE, 53–60.
  * Jabar et al. (2019) Falah Jabar, Joao Ascenso, and Maria Paula Queluz. 2019.  Objective assessment of perceived geometric distortions in viewport rendering of 360° images.  _IEEE Journal of Selected Topics in Signal Processing (JSTSP)_ 14, 1 (2019), 49–63.
  * Kim et al. (2024) Minsu Kim, Jaewon Lee, Byeonghun Lee, Sunghoon Im, and Kyong Hwan Jin. 2024.  Implicit Neural Image Stitching With Enhanced and Blended Feature Reconstruction. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. 4087–4096.
  * Larsen and Fejfar (1974) Max D. Larsen and James L. Fejfar. 1974.  _INTRODUCTION TO GEOMETRY_.  225\.
  * Lee et al. (2022) Jaewon Lee, Kwang Pyo Choi, and Kyong Hwan Jin. 2022.  Learning local implicit fourier representation for image warping. In _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, 182–200.
  * Lee and Jin (2022) Jaewon Lee and Kyong Hwan Jin. 2022.  Local texture estimator for implicit representation function. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1929–1938.
  * Li et al. (2023) Jiacheng Li, Chang Chen, Wei Huang, Zhiqiang Lang, Fenglong Song, Youliang Yan, and Zhiwei Xiong. 2023.  Learning steerable function for efficient image resampling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5866–5875.
  * Li et al. (2024) Runyi Li, Xuhan Sheng, Weiqi Li, and Jian Zhang. 2024.  OmniSSR: Zero-shot Omnidirectional Image Super-Resolution using Stable Diffusion Model.  _arXiv preprint arXiv:2404.10312_ (2024).
  * Li et al. (2022) Weiqi Li, Bin Chen, and Jian Zhang. 2022.  D3c2-net: Dual-domain deep convolutional coding network for compressive sensing.  _arXiv preprint arXiv:2207.13560_ (2022).
  * Liang et al. (2021a) Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021a.  Swinir: Image restoration using swin transformer. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 1833–1844.
  * Liang et al. (2021b) Jingyun Liang, Andreas Lugmayr, Kai Zhang, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2021b.  Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 4076–4085.
  * Lim et al. (2017) Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017.  Enhanced deep residual networks for single image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 136–144.
  * Lin et al. (2023) Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. 2023.  Diffbir: Towards blind image restoration with generative diffusion prior.  _arXiv preprint arXiv:2308.15070_ (2023).
  * Liu et al. (2020) Hongying Liu, Zhubo Ruan, Chaowei Fang, Peng Zhao, Fanhua Shang, Yuanyuan Liu, and Lijun Wang. 2020.  A single frame and multi-frame joint network for 360-degree panorama video super-resolution.  _arXiv preprint arXiv:2008.10320_ (2020).
  * Liu et al. (2021) Qin Liu, Letong Han, Rui Tan, Hongfei Fan, Weiqi Li, Hongming Zhu, Bowen Du, and Sicong Liu. 2021.  Hybrid attention based residual network for pansharpening.  _Remote Sensing_ 13, 10 (2021), 1962.
  * Mitchell and Netravali (1988) Don P Mitchell and Arun N Netravali. 1988.  Reconstruction filters in computer-graphics.  _ACM Siggraph Computer Graphics_ 22, 4 (1988), 221–228.
  * Nishiyama et al. (2021) Akito Nishiyama, Satoshi Ikehata, and Kiyoharu Aizawa. 2021.  360 single image super resolution via distortion-aware network and distorted perspective images. In _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 1829–1833.
  * Ouyang et al. (2022) Hao Ouyang, Tengfei Wang, and Qifeng Chen. 2022.  Restorable image operators with quasi-invertible networks. In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , Vol. 36. 2008–2016.
  * Ozcinar et al. (2019) Cagri Ozcinar, Aakanksha Rana, and Aljosa Smolic. 2019.  Super-resolution of omnidirectional images using adversarial learning. In _2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 1–6.
  * Pan et al. (2022) Zhihong Pan, Baopu Li, Dongliang He, Mingde Yao, Wenhao Wu, Tianwei Lin, Xin Li, and Errui Ding. 2022.  Towards bidirectional arbitrary image rescaling: Joint optimization and cycle idempotence. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17389–17398.
  * Qi et al. (2023) Chenyang Qi, Xin Yang, Ka Leong Cheng, Ying-Cong Chen, and Qifeng Chen. 2023.  Real-time 6K Image Rescaling with Rate-distortion Optimization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 14092–14101.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015.  U-net: Convolutional networks for biomedical image segmentation. In _Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 234–241.
  * Son and Lee (2021) Sanghyun Son and Kyoung Mu Lee. 2021.  Srwarp: Generalized image super-resolution under arbitrary transformation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 7782–7791.
  * Sun and Chen (2020) Wanjie Sun and Zhenzhong Chen. 2020.  Learned image downscaling for upscaling using content adaptive resampler.  _IEEE Transactions on Image Processing (TIP)_ 29 (2020), 4027–4040.
  * Sun et al. (2023) Xiaopeng Sun, Weiqi Li, Zhenyu Zhang, Qiufang Ma, Xuhan Sheng, Ming Cheng, Haoyu Ma, Shijie Zhao, Jian Zhang, Junlin Li, et al. 2023\.  OPDN: Omnidirectional position-aware deformable network for omnidirectional image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW)_. 1293–1301.
  * Sun et al. (2017) Yule Sun, Ang Lu, and Lu Yu. 2017.  Weighted-to-spherically-uniform quality evaluation for omnidirectional video.  _IEEE Signal Processing Letters_ 24, 9 (2017), 1408–1412.
  * Wallace (1991) Gregory K Wallace. 1991.  The JPEG still picture compression standard.  _Commun. ACM_ 34, 4 (1991), 30–44.
  * Wang et al. (2023) Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. 2023.  Exploiting diffusion prior for real-world image super-resolution.  _arXiv preprint arXiv:2305.07015_ (2023).
  * Wang et al. (2024) Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, and Jian Zhang. 2024.  360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2021) Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. 2021.  Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 1905–1914.
  * Wu et al. (2021) Yue Wu, Guotao Meng, and Qifeng Chen. 2021.  Embedding novel views in a single jpeg image. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 14519–14527.
  * Xiao et al. (2012) Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2012.  Recognizing scene viewpoint using panoramic place representation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2695–2702.
  * Xiao et al. (2023) Mingqing Xiao, Shuxin Zheng, Chang Liu, Zhouchen Lin, and Tie-Yan Liu. 2023.  Invertible rescaling network and its extensions.  _International Journal of Computer Vision (IJCV)_ 131, 1 (2023), 134–159.
  * Xiao et al. (2020) Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu. 2020.  Invertible image rescaling. In _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, 126–144.
  * Xie et al. (2021) Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. 2021.  Enhanced invertible encoding for learned image compression. In _Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)_. 162–170.
  * Xing et al. (2024) Qunliang Xing, Mai Xu, Shengxi Li, Xin Deng, Meisong Zheng, Huaida Liu, and Ying Chen. 2024.  Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain.  _arXiv preprint arXiv:2402.17200_ (2024).
  * Xing et al. (2021) Yazhou Xing, Zian Qian, and Qifeng Chen. 2021.  Invertible image signal processing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6287–6296.
  * Xu et al. (2023) Bingna Xu, Yong Guo, Luoqian Jiang, Mianjie Yu, and Jian Chen. 2023.  Downscaled representation matters: Improving image rescaling with collaborative downscaled images. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 12237–12247.
  * Xu et al. (2020a) Mai Xu, Lai Jiang, Chen Li, Zulin Wang, and Xiaoming Tao. 2020a.  Viewport-based CNN: A multi-task approach for assessing 360° video quality.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ 44, 4 (2020), 2198–2215.
  * Xu et al. (2020b) Mai Xu, Chen Li, Shanyi Zhang, and Patrick Le Callet. 2020b.  State-of-the-art in 360 video/image processing: Perception, assessment and compression.  _IEEE Journal of Selected Topics in Signal Processing (JSTSP)_ 14, 1 (2020), 5–26.
  * Yang et al. (2023b) Jinhai Yang, Mengxi Guo, Shijie Zhao, Junlin Li, and Li Zhang. 2023b.  Self-asymmetric invertible network for compression-aware image rescaling. In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , Vol. 37. 3155–3163.
  * Yang et al. (2022a) Li Yang, Mai Xu, Shengxi Li, Yichen Guo, and Zulin Wang. 2022a.  Blind VQA on 360° Video via Progressively Learning From Pixels, Frames, and Video.  _IEEE Transactions on Image Processing (TIP)_ 32 (2022), 128–143.
  * Yang et al. (2022b) Li Yang, Mai Xu, Tie Liu, Liangyu Huo, and Xinbo Gao. 2022b.  TVFormer: Trajectory-guided visual quality assessment on 360° images with transformers. In _Proceedings of the 30th ACM International Conference on Multimedia (ACM MM)_. 799–808.
  * Yang et al. (2023a) Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and Jian Zhang. 2023a.  Implicit neural representation for cooperative low-light image enhancement. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 12918–12927.
  * Yoon et al. (2022) Youngho Yoon, Inchul Chung, Lin Wang, and Kuk-Jin Yoon. 2022.  SphereSR: 360deg Image Super-Resolution With Arbitrary Projection via Continuous Spherical Image Representation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5677–5686.
  * Yu et al. (2023) Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong. 2023.  OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2018a) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018a.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 586–595.
  * Zhang et al. (2024) Xuanyu Zhang, Runyi Li, Jiwen Yu, Youmin Xu, Weiqi Li, and Jian Zhang. 2024.  Editguard: Versatile image watermarking for tamper localization and copyright protection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2022) Xuanyu Zhang, Yongbing Zhang, Ruiqin Xiong, Qilin Sun, and Jian Zhang. 2022.  Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17532–17541.
  * Zhang et al. (2018b) Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018b.  Image super-resolution using very deep residual channel attention networks. In _Proceedings of the European Conference on computer vision (ECCV)_. 286–301.
  * Zhang et al. (2018c) Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018c.  Residual dense network for image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2472–2481.