# L3C-Stereo: Lossless Compression for Stereo Images

Zihao Huang, Zhe Sun, Feng Duan, Andrzej Cichocki, Peiying Ruan, and Chao Li
Z. Huang and F. Duan are with the College of Artificial Intelligence, Nankai
University, Tianjin, China. Z. Sun is with the Computational Engineering
Applications Unit, Head Office for Information Systems and Cybersecurity,
RIKEN, Saitama, Japan. A. Cichocki is with the Skolkovo Institute of Science
and Technology, Moscow, Russia; College of Computer Science, Hangzhou Dianzi
University, China; Department of Informatics, Nicolaus Copernicus University,
Poland and the Systems Research Institute of Polish of Academy of Science,
Warsaw, Poland. P. Ruan is with the NVIDIA AI Technology Center, NVIDIA
Corporation Japan, Japan. C. Li is with the Center for Advanced Intelligence
Project (AIP), RIKEN, Japan. Zihao Huang and Zhe Sun contributed equally to
this workCorresponding author: Feng Duan<EMAIL_ADDRESS>
###### Abstract

A large number of autonomous driving tasks need high-definition stereo images,
which requires a large amount of storage space. Efficiently executing lossless
compression has become a practical problem. Commonly, it is hard to make
accurate probability estimates for each pixel. To tackle this, we propose
L3C-Stereo, a multi-scale lossless compression model consisting of two main
modules: the warping module and the probability estimation module. The warping
module takes advantage of two view feature maps from the same domain to
generate a disparity map, which is used to reconstruct the right view so as to
improve the confidence of the probability estimate of the right view. The
probability estimation module provides pixel-wise logistic mixture
distributions for adaptive arithmetic coding. In the experiments, our method
outperforms the hand-crafted compression methods and the learning-based method
on all three datasets used. Then, we show that a better maximum disparity can
lead to a better compression effect. Furthermore, thanks to a compression
property of our model, it naturally generates a disparity map of an acceptable
quality for the subsequent stereo tasks.

###### Index Terms:

Lossless compression, convolutional neural network, stereo images, disparity
map, arithmetic coding.

##  I Introduction

With the improvement of autonomous driving technologies and the proposal of
superior algorithms, higher quality stereo images are required, which means
that more storage space is needed. Commonly, it requires much more space to
store a lossless image than a lossy one[1]. As a consequence, knowing how to
compress stereo images in a better lossless way has become both essential and
economically important.

Lossless compression has a long history of development. Using this method, it
is possible to reconstruct the stored bitstream into an image that looks
exactly like the original. Although non-learning-based methods such as PNG[2],
WebP[3] and FLIF[4] are widely used, learning-based methods can achieve higher
a compression ratio. Methods [5, 6, 7, 8, 9, 10] all consist of a probability
estimation model and an entropy codec. In the encoding phase, the probability
of all the symbols appearing at each subpixel is modelled, followed by the
images being encoded to a bitstream by the entropy codec. The key to lossless
compression is the accuracy of the probability estimation; the more accurate
it is, the higher the compression ratio achieved is.

Stereo images contain both left and right views captured at the same time,
where similar information naturally exists. Many studies [11, 12, 13, 14, 15]
have leveraged this information estimate disparity map 111A disparity map
refers to the apparent pixel difference or motion between a pair of stereo
images.for downstream tasks. This similar information was first used for deep
lossy compression in [16]. By estimating a disparity map, left view feature
maps were warped to the right ones to achieve a better lossy compression.
However, as far as we know, there is no existing research on lossless
compression for stereo images.

In this paper, we address this challenge and propose a learning-based
architecture for lossless stereo image compression, as illustrated in Fig. 1.
In order to extract useful information from left to right view, a relatively
accurate estimation of the disparity map is needed, since the values in a
disparity map represent the pixel distances in the two view images. Many
studies have shown that hierarchical structures can improve the disparity map
estimation. As a consequence, we adopt the L3C [5], a hierarchically learned
lossless compression structure, as our backbone to then estimate the disparity
map under multi-scaled stereo feature maps to achieve a better right view
compression.

Specifically, given a pair of stereo images $\boldsymbol{X}_{L}$ and
$\boldsymbol{X}_{R}$, we employ $S$ encoders $\texttt{E}^{(s)}$ for feature
extraction, yielding the multi-scale auxiliary features
$\boldsymbol{Z}_{L}^{(s)}$ and $\boldsymbol{Z}_{R}^{(s)}$ which will be
encoded to a bitstream and transferred to the receiver. Then, $S$ decoders
$\texttt{D}^{(s)}$ are employed to integrate current scale features with lower
scale features. For the left view, the decoded feature maps are used to
predict the probability distributions $p$ for $\boldsymbol{X}_{L}$ and
$\boldsymbol{Z}_{L}^{(s)}$. For the right view, the disparity map is leveraged
to flow information from the left view to the right view. After that, the
fine-tuned decoded feature maps allow us to get a more accurate probability
distribution for the right view. In the end, the sender uses a codec to encode
all the images and auxiliary features to a bitstream. Regarding the receiver,
the codec needs the same probability distributions $p$ to decode the
bitstream. Consequently, it is necessary to reconstruct the distributions at
the receiver. Starting with the smallest scale auxiliary feature
$\boldsymbol{Z}_{L}^{(S)}$ and $\boldsymbol{Z}_{R}^{(S)}$, we assume the prior
distribution to be uniform both in the sender and the receiver. This allows
the codec to decode $\boldsymbol{Z}_{L}^{(S)}$ and $\boldsymbol{Z}_{R}^{(S)}$
in a non-optimal way, with a low cost due to its low scale. Finally, all the
images and features can be decoded from a low to a high scale.

The proposed algorithm is evaluated on the KITTI 2012/2015 and Scene Flow
datasets. Experiments show that our method outperforms the baseline and other
lossless image compression methods. Furthermore, the effects on compression by
the supervised/self-supervised disparity map estimation and by the maximum
disparity are also investigated. Finally, note that the disparity map which
made as a byproduct of our method can be easily used for the follow-up
automatic driving tasks.

##  II Related Works

###  II-A Learned lossy compression

Lossy compression [17] has a higher compression ratio, but the resulting image
quality is poorer. Given an image $\boldsymbol{x}$, an encoder and a quantizer
are used to generate a discrete representation $\boldsymbol{y}$, which will be
encoded to a bitstream for transfer. Then, a decoder will be used to generate
an output $\widehat{\boldsymbol{x}}$ which is similar to $\boldsymbol{x}$. In
the training phase, it is hoped that this model can achieve a high compression
ratio while being able to produce high quality images. Considering the trade-
off, the loss function is usually of this form:

$L=l(\boldsymbol{x},\widehat{\boldsymbol{x}})+\beta R(\boldsymbol{y}).$ (1)

Where $l(\cdot)$ is the reconstruction error and $R(\boldsymbol{y})$ is the
cost of encoding $\boldsymbol{y}$ to a bitstream. Ballé et al.[18, 19, 20]
proposed the generalized divisive normalization (GDN) and several CNN-based
variational autoencoders for learned lossy compression, introducing the side
information to obtain a better reconstruction. Chen et al.[21] used self-
attention to improve the compression effect. References [22, 23, 24]
implemented lossy compression using RNN. Liu et al.[16] also made use of the
side information for stereo image lossy compression, but the information was
from the left view image rather than from the extracted representation.

###  II-B Learnable lossless compression

Lossless compression does not usually have a high compression ratio, because
high frequency components or noise in the image are difficult to compress. All
the proposed methods need to estimate a distribution on the original image
$\boldsymbol{x}$. The pixel-wise auto-regressive model explored in [25, 26,
27] predicts the distribution of each pixel in turn conditionally:

$p\left(\boldsymbol{x}\right)=\prod_{i}p\left(\boldsymbol{x}_{i}|\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{i-1}\right),$
(2)

where $i$ is the pixel index. The multi-scale auto-regressive model [6, 5, 10,
28] estimates the distribution of pixels jointly based on low-scale auxiliary
representations:

$p\left(\boldsymbol{x}_{0},\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{N}\right)=\prod_{n=0}^{N-1}p\left(\boldsymbol{x}_{n}|\boldsymbol{x}_{n+1},\boldsymbol{x}_{n+2},...,\boldsymbol{x}_{N}\right)\cdot
p\left(\boldsymbol{x}_{N}\right),$ (3)

where $\boldsymbol{x}_{0}$ is the original image while the other
$\boldsymbol{x}_{n}$ are the low-scale auxiliary representations. Such methods
are generally much faster than the previous methods, benefiting from a joint
estimation rather than a conditional estimation. Flow-based models[8, 9] learn
an invertible mapping function $f(\cdot)$ from image space $\mathbb{X}$ to
latent space $\mathbb{Z}$, where the variables follow a predefined
distribution $p_{\boldsymbol{z}}$:

$p\left(\boldsymbol{x}\right)=p_{\boldsymbol{z}}\left(f\left(\boldsymbol{x}\right)\right)\left|det\left(\frac{\partial
f\left(\boldsymbol{x}\right)}{\partial\boldsymbol{x}}\right)\right|.$ (4)

However, this kind of method is more suitable for low-resolution images.
Moreover, some studies [7, 29] have employed lossy compression and residual
lossless compression while achieving competitive results.

###  II-C Stereo matching

A variety of studies have been done on stereo matching based on deep learning,
and all of them can be roughly grouped into two types of network structures
and two types of training methods. For the structure type, one [30, 31, 32,
33] uses the 2D-CNN-based encoder-decoder to compute the correlation scores
between the left and right view features, while the other[11, 12, 13, 14, 15]
uses 3D-CNN to estimate the disparity from a 4D cost volume. As for the
training methods, most studies have chosen to use supervised learning, given a
ground-truth disparity map as a label. However, the ground-truth disparity is
not always available. Hence, the other method [34, 35] is self-supervised.
Here, the network needs to warp the left view to the right view based on the
estimated disparity map while minimising the reconstruction error. The
disadvantage of this method is that different homologous pixel values and
occlusion areas will lead to larger estimation errors. We will show the effect
of the model under different supervised methods.

##  III Methods

Figure 1: An overview of the L3C-Stereo architecture. $\texttt{E}^{(s)}$ and
$\texttt{D}^{(s)}$ are the encoder and decoder modules, respectively. The left
and right view images $\boldsymbol{X}_{L}$ and $\boldsymbol{X}_{R}$ share the
same $\texttt{E}^{(s)}$ and $\texttt{D}^{(s)}$. Q is the quantization module,
which is a non-learning module. The $\boldsymbol{Z}^{s}$s are the auxiliary
feature maps in different scales. Module $\texttt{Warp}^{s}$ receives the
$\boldsymbol{f}^{(s)}$ from the left and right view feature maps and the
disparity map from $\texttt{Warp}^{(s+1)}$. It then estimates a disparity map
for the right view, and finally it warps $\boldsymbol{Z}_{L}^{(s-1)}$ into
$\overline{\boldsymbol{Z}}_{R}^{(s-1)}$, which is similar to
$\boldsymbol{Z}_{R}^{(s-1)}$. $\texttt{P}^{(s)}$ is the probability estimation
module.

###  III-A Lossless compression

Commonly, given a stream of symbols $x_{1},x_{2}...x_{N}$, the value of each
symbol $x_{i}$ is obtained from a set
$\mathcal{S}=\left\\{0,1,2,...,S\right\\}$ ($S=255$ for image symbol) based on
a probability mass function $\widetilde{p}$. Knowing this function
$\widetilde{p}$, we can encode the stream into a bitstream, and the entropy of
$\widetilde{p}$, which equals the number of bits required to encode $x_{i}$,
is

$H\left(\widetilde{p}\right)=\mathbb{E}_{x_{i}\sim\widetilde{p}}\left[-log_{2}\widetilde{p}\left(x_{i}\right)\right].$
(5)

However, $\widetilde{p}$ is not known, and we can only estimate a $p$ function
that is close to $\widetilde{p}$, achieving a bitcost of:

$H\left(\widetilde{p},p\right)=\mathbb{E}_{x_{i}\sim\widetilde{p}}\left[-log_{2}p\left(x_{i}\right)\right].$
(6)

As a simple example, we can assume that we have a pixel $x$ with a value of
20. Since we know exactly what the value of the pixel is, as a result
$\widetilde{p}_{x}(x=20)=1$ and $\widetilde{p}_{x}(x\neq 20)=0$. With this
prior knowledge, in fact no bits are required to encode $x$. If we get a
$p_{x}$ obeying a uniform distribution, then
$H\left(\widetilde{p}_{x},p_{x}\right)=-log_{2}\left(1/255\right)=7.99$, which
means that using $p_{x}$ to encode $x$ the average bitcost is 7.99.
Consequently, the more accurate the probability mass function used is, the
higher the compression ratio of the model will be.

Arithmetic Coding (AC) is one of the main algorithms used in lossless image
compression, and it is a form of entropy encoding. In simple terms, based on a
given probability mass function, AC divides the interval $[0,1)$ into multiple
subintervals. Each subinterval represents a symbol, and the size of the
subinterval is proportional to the probability of this symbol appearing. In
the beginning, $L$ is set to 0 and $H$ is set to 1. Then, the symbols in the
stream are read in turn, updating the upper and lower bounds of the
subinterval $[L,H)$. Finally, any decimal in the obtained subinterval is
outputted in binary form to obtain the bitstream. In this study, Adaptive
Arithmetic Coding (AAC) was used to encode the image stream, since there is a
high correlation among nearby pixels and thus an adaptive form can achieve a
better compression.

Figure 2: Architecture details for $\texttt{E}^{(s)}$, $\texttt{D}^{(s)}$ and
$\texttt{P}^{(s)}$. The contents within the dashed rectangles are the details
of the respective modules or blocks indicated above them. All the black thin
rectangles represent a 2D Conv. layer. Their heights represent the input size
and the output channels are $64$ except for the one before module Q. The green
thin rectangle is a pixel shuffling upsampling. The three yellow thin
rectangles are 2D Conv. layers with different dilation rates. Their outputs
are concatenated together and are sent to the next Conv. layer.

###  III-B L3C-Stereo Overview

Fig. 1 shows the massive proposed structure. The whole model focuses on
predicting a distribution for each subpixel222A subpixel is the smallest unit
of an image in a single channel. A pixel contains three subpixels for RGB
images.. Instead of directly predicting the distribution on
$\boldsymbol{X}_{L}$ and $\boldsymbol{X}_{R}$, several auxiliary feature maps
are used to simplify this task. Fig. 2 shows an encoder-decoder flow and the
probability estimation module under scale $s$. Assuming that the
$\boldsymbol{X}_{L}$ and $\boldsymbol{X}_{R}$ get a shape of $3\times H\times
W$, after $S$ encoder modules we get auxiliary feature maps
$\boldsymbol{Z}_{L}^{(s)}$ and $\boldsymbol{Z}_{R}^{(s)}$, where $s=1,2...S$,
with a shape of $C\times H/2^{s}\times W/2^{s}$ ($C=5$ in all models appearing
in this paper).

In this model, we jointly predict:

$\displaystyle
p(\boldsymbol{X}_{L},\boldsymbol{Z}_{L}^{(1)},...,\boldsymbol{Z}_{L}^{(S)})=$
(7) $\displaystyle\qquad
p(\boldsymbol{X}_{L}\mid\boldsymbol{Z}_{L}^{(1)},...,\boldsymbol{Z}_{L}^{(S)})\prod_{s=1}^{S}p(\boldsymbol{Z}_{L}^{(s)}\mid\boldsymbol{Z}_{L}^{(s+1)},...,\boldsymbol{Z}_{L}^{(S)}),$

$\displaystyle
p(\boldsymbol{X}_{R},\boldsymbol{Z}_{R}^{(1)},...,\boldsymbol{Z}_{R}^{(S)}\mid\overline{\boldsymbol{X}}_{R},\overline{\boldsymbol{Z}}_{R}^{(1)},...,\overline{\boldsymbol{Z}}_{R}^{(S-1)})=$
(8) $\displaystyle\qquad
p(\boldsymbol{X}_{R}\mid(\boldsymbol{Z}_{R}^{(1)},\overline{\boldsymbol{X}}_{R}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)}))\cdot$
$\displaystyle\qquad\prod_{s=1}^{S-1}p(\boldsymbol{Z}_{R}^{(s)}\mid(\boldsymbol{Z}_{R}^{(s+1)},\overline{\boldsymbol{Z}}_{R}^{(s)}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)}))p(\boldsymbol{Z}_{R}^{(S)}).$

Where $\overline{\boldsymbol{X}}_{R}$ and
$\overline{\boldsymbol{Z}}_{R}^{(s)}$ are the known from the Warp modules, and
$p(\boldsymbol{Z}_{L}^{(S)})$ and $p(\boldsymbol{Z}_{R}^{(S)})$ follow a
uniform distribution.

Algorithm pipeline: Taking two view images as inputs, the L3C-Stereo method
employs three encoder modules to extract the multi-scaled feature
representation $\widetilde{\boldsymbol{Z}}^{(s)}$. A quantization module is
used to discrete the outputs, so that the value of
$\widetilde{\boldsymbol{Z}}^{(s)}$ can be transformed to
$\boldsymbol{Z}^{(s)}$(which can take values of 0 255, just like an RGB
image). Then, three decoder modules are used to generate feature maps
$\boldsymbol{f}^{(s)}$ for the probability estimation. Until this point, the
structure is very similar to the U-Net and Pyramid Network, except for the use
of the quantization module which would lead to information loss. After that,
for the left view, the process is relatively simple. Receiving
$\boldsymbol{f}_{L}^{(s)}$, module $\texttt{P}_{L}^{(s)}$ predicts a
conditional distribution
$p(\boldsymbol{Z}_{L}^{(s-1)}\mid\boldsymbol{f}_{L}^{(s)})$. Since
$\boldsymbol{f}_{L}^{(s)}$ contains the information from
$\boldsymbol{Z}_{L}^{(s)},\boldsymbol{Z}_{L}^{(s+1)},...,\boldsymbol{Z}_{L}^{(S)}$,
this conditional distribution is equal to
$p(\boldsymbol{Z}_{L}^{s-1}\mid\boldsymbol{Z}_{L}^{(s)},\boldsymbol{Z}_{L}^{(s+1)},...,\boldsymbol{Z}_{L}^{(S)})$.
For the right view, $\boldsymbol{f}_{L}^{(s)}$ and $\boldsymbol{f}_{R}^{(s)}$
are employed to generate the cost volume, which is then taken by the
$\texttt{Warp}^{(s)}$ module, along with the aggregated cost volume from
$\texttt{Warp}^{(s+1)}$, to estimate the disparity map at this scale. Based on
that, we warp $\boldsymbol{Z}_{L}^{(s-1)}$ into
$\overline{\boldsymbol{Z}}_{R}^{(s-1)}$, expecting
$\overline{\boldsymbol{Z}}_{R}^{(s-1)}$ and $\boldsymbol{Z}_{R}^{(s-1)}$ to be
highly similar. Conducting Conv. $1\times 1$ for the fusion between
$\boldsymbol{f}_{R}^{(s)}$ and $\overline{\boldsymbol{Z}}_{R}^{(s-1)}$, we
generate $\overline{\boldsymbol{f}}_{R}^{(s)}$, which is the fine-tuned
version of $\boldsymbol{f}_{R}^{(s)}$. Finally, the same operation as with the
left view is used to estimate the conditional probability distribution.

Figure 3: Architecture details for $\texttt{Warp}^{(s)}$. $\downarrow/\uparrow
n\texttt{x}$ represents the interpolation with $n$ times under a bilinear or
trilinear mode. The purple arrows represent the inputs for
$\texttt{Warp}^{(s)}$ and the output for $\texttt{Warp}^{(s-1)}$. “C” refers
to the concatenate operation. The slender cube represents a 3D Conv. layer,
and the “Aggregation” structure is similar to $\texttt{P}^{(s)}$.

###  III-C Quantization

To discrete the value $\widetilde{z}\in\widetilde{\boldsymbol{Z}}^{(s)}$, we
use the scalar quantization approach proposed in [36]. Based on a list of
quantized scalars
$\mathbb{L}=\left\\{l_{1},l_{2},...,l_{L}\right\\}\subset\mathbb{R}$, the hard
manner directly uses the nearest neighbor assignment algorithm:

$\displaystyle
z=Q_{h}(\widetilde{z})=\mathbb{L}\left[argmin_{j=1}^{L}\left|l_{j}-\widetilde{z}\right|\right].$
(9)

However, this manner is non-differentiable. A soft manner solves this problem:

$\displaystyle
z=Q_{s}(\widetilde{z})=\sum_{j=1}^{L}\frac{exp(-\sigma_{q}\left|l_{j}-\widetilde{z}\right|)}{\sum_{k=1}^{L}exp(-\sigma_{q}\left|l_{k}-\widetilde{z}\right|)}l_{j},$
(10)

where $\sigma_{q}$ is a softness hyper-parameter. During the training phase,
we use $Q_{h}$ for the forward propagation and $Q_{s}$ for the backward
propagation:

$\displaystyle
z=\left\\{Q_{h}(\widetilde{z})-Q_{s}(\widetilde{z})\right\\}_{require\\_no\\_grad}+Q_{s}(\widetilde{z}).$
(11)

In this paper, we set $L=25$ and evenly distributed $\mathbb{L}$ in $[-1,1]$.

###  III-D Warp

A disparity map refers to the apparent pixel difference or motion between a
pair of stereo images. As a result, we can move the left view pixel to the
position where the right view pixel may appear based on the disparity in a
pixel-wise manner. Then, the reconstructed right view can use the model to
predict the distribution of the real right view pixel. To reduce the search
space, the disparity is only estimated in the width dimension, and we use the
sliding window on the width rather than on the full line data.

Specifically, assuming the disparity maximum, a.k.a. sliding widow, is $D$
pixels. The cost volume is generated by comparing the feature maps of both
views with $D$ times of horizontal shift. Given $\boldsymbol{f}_{L}^{(s)}$ and
$\boldsymbol{f}_{R}^{(s)}$ with a shape of $C\times H\times W$, the cost
volume $CV$ can be written as (in a numpy form):

$\displaystyle
CV=\begin{bmatrix}\left\|\boldsymbol{f}_{L}^{(s)}-\boldsymbol{f}_{R}^{(s)}\right\|&&&\\\
\left\|\boldsymbol{f}_{L}^{(s)}[1:]-\boldsymbol{f}_{R}^{(s)}[:-1]\right\|&\boldsymbol{O}&&\\\
\left\|\boldsymbol{f}_{L}^{(s)}[2:]-\boldsymbol{f}_{R}^{(s)}[:-2]\right\|&\boldsymbol{O}&\boldsymbol{O}&\\\
\vdots&\vdots&\vdots&\vdots\\\
\left\|\boldsymbol{f}_{L}^{(s)}[D-1:]-\boldsymbol{f}_{R}^{(s)}[:1-D]\right\|&\boldsymbol{O}&...&\boldsymbol{O}\end{bmatrix}.$
(12)

The $CV$ is a 4D tensor with a shape of $C\times D\times H\times W$. Given an
exact pixel position, we will see a vector with $D$ numbers, and the smaller
this number is, the more likely that the index of this number is the distance
between the two views of the corresponding pixel. After that, an aggregation
block is introduced for refining the cost volume among all channels $C$,
generating a cost volume with a shape of $D\times H\times W$. Then, by
conducting a softmax operation along $D$ dimension we will get a regularized
cost volume, where a fiber along $D$ dimension represents the probability that
this pixel in the right view matches all the pixels in the sliding window of
the left view. Subsequently, the left view pixels are moved to the right view
pixels in a soft manner: Given a pixel $\boldsymbol{z}_{R}$, a fiber
$\boldsymbol{d}$ is extracted. Starting at this pixel, we then take D pixels
$\boldsymbol{z}_{L}$ from the left view along the horizontal direction and
apply the following equation:

$\displaystyle\overline{z}_{R}=sum(\boldsymbol{b}\cdot\boldsymbol{z}_{L}).$
(13)

In the end, the warped $\overline{\boldsymbol{Z}}_{R}^{(s)}$ is simply
concatenated with $\boldsymbol{f}_{R}^{(s)}$ and the two are fused with
several 2D conv. layers.

Considering the fact that the proposed model naturally generates multi-scaled
feature maps, note that better results can be obtained by using the feature
map of the previous scale to assist the module of the current scale in the
prediction. Therefore, we introduce $\boldsymbol{f}_{cv}^{(s+1)}$ into
$\texttt{Warp}^{(s)}$ before aggregation.

###  III-E Logistic Mixture Model

In the prediction phase, a mixture of discrete logistics with $K$ components
are used [5, 27] to model the probability mass function of each subpixel in
$\boldsymbol{X}$ and $\boldsymbol{Z}^{(s)},s=1,2...,S$. Since the left and
right views are handled in the same way, here the L and R subscripts will be
omitted.

A single logistic distribution is given by:

$\displaystyle
p_{l}(z|\mu,\sigma)=\frac{e^{-(z-\mu)/\sigma}}{\sigma(1+e^{-(z-\mu)/\sigma})^{2}}.$
(14)

For the discrete form, we take advantage of its cumulative distribution
function to compute the cumulative probability of the discrete interval:

$\displaystyle
p_{l}(z|\mu,\sigma)=sigmoid(\frac{z+b/2-\mu}{\sigma})-sigmoid(\frac{z-b/2-\mu}{\sigma}),$
(15)

where $b$ is the discrete interval, $b=1$ for $X$ and $b=1/12$ for $Z^{(s)}$.

Given a pixel $\boldsymbol{x}$ from $\boldsymbol{X}$, where
$\left(x_{i},i=1,2,3\right)\in\boldsymbol{X}$, it is modeled in an auto-
regression form:

$\displaystyle p(\boldsymbol{x}|\boldsymbol{f}^{(1)})=$ $\displaystyle
p(x_{1},x_{2},x_{3}|\boldsymbol{f}^{(1)})$ (16) $\displaystyle=$
$\displaystyle p_{m}(x_{1}|\boldsymbol{f}^{(1)})\cdot
p_{m}(x_{2}|\boldsymbol{f}^{(1)},x_{1})\cdot
p_{m}(x_{3}|\boldsymbol{f}^{(1)},x_{1},x_{2}).$

Where $p_{m}$ is a mixture of logistic distributions $p_{l}$. $p_{m}$ contains
$K$ components, and the $k$th component has three parameters: weight
$\pi_{i}^{k}$, mean $\mu_{i}^{k}$ and variance $\sigma_{i}^{k}$. Then, we get:

$\displaystyle
p_{m}(x_{1}|\boldsymbol{f}^{(1)})=\sum_{k}\pi_{1}^{k}p_{l}(x_{1}|\mu_{1}^{k},\sigma_{1}^{k})$
(17) $\displaystyle
p_{m}(x_{2}|\boldsymbol{f}^{(1)},x_{1})=\sum_{k}\pi_{2}^{k}p_{l}(x_{2}|\widetilde{\mu}_{2}^{k},\sigma_{2}^{k})$
$\displaystyle
p_{m}(x_{3}|\boldsymbol{f}^{(1)},x_{1},x_{2})=\sum_{k}\pi_{3}^{k}p_{l}(x_{3}|\widetilde{\mu}_{3}^{k},\sigma_{3}^{k}).$

As previously mentioned, on the topic of auto-regression, the model provides
the coefficients
$\lambda_{\alpha}^{k},\lambda_{\beta}^{k},\lambda_{\gamma}^{k}$ for a better
prediction of $\mu$:

$\displaystyle\widetilde{\mu}_{2}^{k}$
$\displaystyle=\mu_{2}^{k}+\lambda_{\alpha}^{k}x_{1}$ (18)
$\displaystyle\widetilde{\mu}_{3}^{k}$
$\displaystyle=\mu_{2}^{k}+\lambda_{\beta}^{k}x_{1}+\lambda_{\gamma}^{k}x_{2}.$

Given a pixel $\boldsymbol{z}^{(s)}$ from $\boldsymbol{Z}^{(s)}$ under scale
$s$, where $\left(z_{i}^{(s)},i=1,2,3,4,5\right)\in\boldsymbol{Z}^{(s)}$, we
get

$\displaystyle p(\boldsymbol{z}^{(s)}|\boldsymbol{f}^{(s+1)})$
$\displaystyle=p(z_{1}^{(s)},z_{2}^{(s)},z_{3}^{(s)},z_{4}^{(s)},z_{5}^{(s)}|\boldsymbol{f}^{(s+1)})$
(19) $\displaystyle=\prod_{i=1}^{5}p_{m}(z_{i}^{(s)}|\boldsymbol{f}^{(s+1)})$

and

$\displaystyle
p_{m}(z_{i}^{(s)}|\boldsymbol{f}^{(s+1)})=\sum_{k}\pi_{i}^{k(s)}p_{l}(z_{i}^{(s)}|\mu_{i}^{k(s)},\sigma_{i}^{k(s)}).$
(20)

We do not auto-regress $\mu$ when predicting the distribution of $Z^{(s)}$.

All of $\mu$, $\pi$, $\sigma$ and $\lambda$ (if needed) are obtained using the
last 2D Conv. $1\times 1$ layer of module P. Then, by carrying out the above
operation, we will get the probability tensor with a shape of $3\times H\times
W\times 256$ for RGB images and $5\times H^{(s)}\times W^{(s)}\times L$ for
$\boldsymbol{Z}^{(s)}$.

###  III-F Loss

Equation 6 describes the difference between the two distributions. Because the
model gives the probability of every value of each subpixel, the subpixel
value can be changed to a one-hot form as the target distribution. Then, the
loss function is obtained as follows:

$\displaystyle\mathcal{L}$
$\displaystyle=H(\widetilde{p}(\boldsymbol{X}_{L}),p(\boldsymbol{X}_{L}\mid\boldsymbol{Z}_{L}^{(1)},...,\boldsymbol{Z}_{L}^{(S)}))+$
(21)
$\displaystyle\sum_{s=1}^{S-1}H(\widetilde{p}(\boldsymbol{Z}_{L}^{(s)}),p(\boldsymbol{Z}_{L}^{(s)}\mid\boldsymbol{Z}_{L}^{(s+1)},...,\boldsymbol{Z}_{L}^{(S)}))+$
$\displaystyle
H(\widetilde{p}(\boldsymbol{X}_{R}),p(\boldsymbol{X}_{R}\mid(\boldsymbol{Z}_{R}^{(1)},\overline{\boldsymbol{X}}_{R}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)})))+$
$\displaystyle\sum_{s=1}^{S-1}H(\widetilde{p}(\boldsymbol{Z}_{R}^{(s)}),p(\boldsymbol{Z}_{R}^{(s)}\mid(\boldsymbol{Z}_{R}^{(s+1)},\overline{\boldsymbol{Z}}_{R}^{(s)}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)})))$
$\displaystyle=-(log_{2}(p(\boldsymbol{X}_{L}\mid\boldsymbol{Z}_{L}^{(1)},...,\boldsymbol{Z}_{L}^{(S)}))+$
$\displaystyle\sum_{s=1}^{S-1}log_{2}(p(\boldsymbol{Z}_{L}^{(s)}\mid\boldsymbol{Z}_{L}^{(s+1)},...,\boldsymbol{Z}_{L}^{(S)}))+$
$\displaystyle
log_{2}(p(\boldsymbol{X}_{R}\mid(\boldsymbol{Z}_{R}^{(1)},\overline{\boldsymbol{X}}_{R}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)})))+$
$\displaystyle\sum_{s=1}^{S-1}log_{2}(p(\boldsymbol{Z}_{R}^{(s)}\mid(\boldsymbol{Z}_{R}^{(s+1)},\overline{\boldsymbol{Z}}_{R}^{(s)}),...,(\boldsymbol{Z}_{R}^{(S)},\overline{\boldsymbol{Z}}_{R}^{(S-1)})))).$

TABLE I: Comparison. “bpsp” means bits per subpixel. The bitcost without compression is 8 bits. bpsp | KITTI2012 | KITTI2015 | Flythings3D
---|---|---|---
Left view | Right view | All view | Left view | Right view | All view | Left view | Right view | All view
PNG | 4.639 | 4.617 | 4.628 | 4.706 | 4.484 | 4.595 | 3.721 | 3.721 | 3.721
WebP | 4.156 | 4.018 | 4.087 | 4.062 | 3.797 | 3.929 | 2.735 | 2.736 | 2.736
FLIF | 4.026 | 3.884 | 3.955 | 4.077 | 3.806 | 3.941 | 2.601 | 2.602 | 2.602
L3C | 3.946 | 3.803 | 3.875 | 3.611 | 3.339 | 3.475 | 2.587 | 2.589 | 2.588
L3C-Stereo | 3.946 | 3.727 | 3.837 | 3.611 | 3.237 | 3.424 | 2.587 | 2.297 | 2.442

##  IV Experiments

In this section, we will aim to demonstrate that the L3C-Stereo technique
outperforms all the compared methods, including PNG, WebP, FLIF and L3C.
Ablation studies on supervised/self-supervised disparity map estimation, on
the maximum disparity and on the stereo matching structure are conducted to
explore the influence of network hyper-parameters on the method’s performance.

###  IV-A Experiment Setup

Datasets The methods are evaluated using three datasets:

  1. 1.

KITTI2012: A real-world dataset with images collected using two high-
resolution colour video cameras from a standard station wagon. It consists of
388 training image pairs and 390 testing image pairs, each one of size
$376\times 1240$.

  2. 2.

KITTI2015[37]: A real-world dataset with images collected using two high-
resolution colour video cameras from a standard station wagon. It consists of
400 training image pairs and 400 testing image pairs, each one of size
$376\times 1240$.

  3. 3.

Scene Flow[30]: A large synthetic dataset. One of its subsets, FlyThings3D,
contains 22,390 training image pairs and 4,370 testing image pairs, each one
of size $540\times 960$. This subset was picked to be used as our dataset
rather than the full Scene Flow dataset, since it was the only one out of the
subsets in Scene Flow containing a testing set.

Training The proposed model was implemented with Pytorch[38]. All models were
optimized with RMSProp[39], with a batch size of 2. The images were randomly
cropped to $128\times 512$. The initial learning rate was 1e-4 and decayed by
a factor of 0.75 every 400 epochs for the KITTI2012/2015 datasets and every 10
epochs for the Scene Flow dataset. For a fair comparison with L3C, we firstly
trained L3C following the same network setup used in [5], and the learning
rate decayed by a factor of 0.75 every 1500 epochs for the KITTI2012/2015
datasets and every 30 epochs for the Scene Flow dataset. After L3C converged
completely, the weights $\texttt{E}^{(s)}$, $\texttt{D}^{(s)}$ and
$\texttt{P}_{L}^{(s)}$($s=1,2,3$) were replaced into some modules in
L3C-Stereo and were frozen. The default maximum disparity was 64. All models
is trained and tested on NVIDIA RTX 3090 unless otherwise specified.

###  IV-B Comparison

The compared methods include PNG, WebP, FLIF and L3C. Results are shown in
Table I. The evaluation criterion used is bit per sub-pixel (bpsp). Since part
of the weights used in the training of L3C-Stereo came from the trained L3C,
the evaluation results of the L3C and L3C-Stereo left view were the same. For
the right view, the proposed method outperforms all other compared methods,
including L3C, where L3C-Stereo outperforms it by a small margin of $2.25\%$
on KITTI2012 and $3.15\%$ on KITTI2015, and by a larger margin of $12.7\%$ on
the Scene Flow dataset.

Fig. 4 gives several qualitative results with regards to the disparity
prediction and the reconstructed right view. The samples were randomly chosen
from the testing set. Table II gives the image quality of the reconstructed
right view (PSNR, SSIM) and the quality of the disparity map ($>$3px, EPE).
“$>$3px” represents the percentage of pixels for which the disparity error is
more than three pixels (or equivalently, more than $5\%$) from all testing
images. End-point error (EPE) is the mean absolute disparity error in pixels.
From the results, it can be seen that the reconstructed right view from the
real-world image has a higher PSNR and SSIM than the one from the synthetic
image, but nevertheless the compression effect is not significantly improved.
We think that real-world images have gentler pixel features, resulting in a
gentler distribution of pixels predicted by the network. Since the KITTI
datasets do not provide a right view disparity map, the quality of the
disparity prediction cannot be calculated. For the disparity error, the
results on the Scene Flow dataset are unexpected. More than $30\%$ of the
image area has an error of more than 3 pixels, as can be seen from the
relatively blurry reconstruction of nearby objects in Fig. 4.(c). The
disparity of these objects exceeds our default 64 pixels, which leads to the
above result. After using a larger maximum disparity, this phenomenon is
alleviated.

Figure 4: Qualitative results from L3C-Stereo. TABLE II: quantitative results from L3C-Stereo. | PSNR | SSIM | >3px(%) | EPE
---|---|---|---|---
KITTI 2012 | 21.23 | 0.9225 | - | -
KITTI 2015 | 22.73 | 0.9531 | - | -
Scene Flow | 19.01 | 0.7231 | 31.80 | 21.08

###  IV-C Ablation study

In this experiment, the impact of the additional hyper-parameters on
L3C-Stereo was evaluated on the Scene Flow dataset, since it is the only
dataset which could provide a right view disparity map. All training settings
remained the same as those in Section IV-B.

For the disparity estimation, current studies are based on two approaches to
train the networks; one involves using a ground-truth disparity map for
supervised training, while the other relies on minimising the warping error of
the reconstructed view. L3C-Stereo adopts the second method, which doesn’t
require the use of any ground-truth disparity map. To discuss whether the
supervised method can obtain a better performance, a term is added to the loss
function:

$\displaystyle Loss=$
$\displaystyle\mathcal{L}+\lambda(\frac{\alpha_{0}}{N}\sum_{n=1}^{N}\left\|d_{n}-\widehat{d}_{n}^{0}\right\|_{1}+$
(22)
$\displaystyle\frac{\alpha_{1}}{N}\sum_{n=1}^{N}\left\|d_{n}-\widehat{d}_{n}^{1}\right\|_{1}+\frac{\alpha_{2}}{N}\sum_{n=1}^{N}\left\|d_{n}-\widehat{d}_{n}^{2}\right\|_{1})$

where $N$ is the number of pixels, $\lambda$ is a trade-off parameter and we
set $\lambda=0.01$, $\alpha_{0}=1.0$, $\alpha_{1}=0.5$, $\alpha_{2}=0.25$.
$d_{n}$ is the ground-truth disparity, $\widehat{d}_{n}^{s}(s=0,1,2)$ is the
predicted disparity under three scale. Before calculating, the low-scale
predicted disparity map is up-sampled in bilinear mode so as to be of the same
size as the ground-truth disparity map.

Other three ablation settings include:

  1. 1.

Disabling the warping function and replacing the reconstructed right view
image with the left view image.

  2. 2.

Disabling $\boldsymbol{f}_{cv}^{(s)}$ from module $\texttt{Warp}^{(s)}$ and
doubling the output channels to be consistent with the previous layer.

  3. 3.

The options for maximum disparity include 64, 128 and 192.

TABLE III: Ablation Study on L3C-Stereo. Disparity estimation | Ablation | D_max | bpsp | PSRN | SSIM | >3px(%) | EPE
---|---|---|---|---|---|---|---
- | Baseline | - | 2.589 | - | - | - | -
- | Use left view | - | 2.539 | - | - | - | -
Self-supervised | Disable $\boldsymbol{f}_{cv}$ | 64 | 2.306 | 19.53 | 0.7413 | 32.10 | 21.08
Default | 64 | 2.296 | 19.01 | 0.7231 | 31.80 | 21.08
128 | 2.231 | 21.96 | 0.9110 | 19.41 | 12.94
192 | 2.226 | 21.39 | 0.8959 | 17.79 | 13.19
Supervised | Default | 64 | 2.310 | 18.36 | 0.7163 | 24.56 | 19.76
128 | 2.251 | 19.89 | 0.8623 | 9.89 | 9.73
196 | 2.236 | 19.83 | 0.8663 | 7.95 | 7.95

All the results are presented in Table III. “Baseline” refers to the L3C
result, using none of the disparity or left view information. “Use left view”
corresponds to point 1 above. “Disable $\boldsymbol{f}_{cv}$” corresponds to
point 2 above. “D_max” refers to maximum disparity. “bpsp” means bits per sub-
pixel for the right view images. “PSNR” and “SSIM” evaluate the quality of the
reconstructed right view. Finally, “$>$3px(%)” and “EPE” evaluate the quality
of the disparity map. Fig. 5 shows the disparity map and the reconstructed
right view from the two cases, in which one is easy to compress and the other
is hard to compress. In the figure, “S” means “Supervised” and “SS” means
“Self-supervised”. The reconstructed ground-truth (GT) is the result obtained
by warping the left view using the ground-truth disparity map, where you may
notice some ghosting in the image. “bpsp” is the compressed right view result
based on the shown reconstructed right view. The brighter the disparity map,
the greater the disparity and therefore the closer the object is.

TABLE IV: Comparison between the different disparity estimation methods. |  $>$3px(%) | EPE
---|---|---
L3C-Stereo | 7.95 | 7.95
GC-Net[11] | 9.34 | 2.51
PSMNet[12] | 3.80 | 1.09
GwcNet[15] | 3.30 | 0.765

Results show that all the L3C-Stereo structures outperform, by a large margin,
those which disable the left view or disparity information. Compared with the
baseline, directly using a left view image only leads to a small increase in
performance. Since there is a pixel level difference between the two views,
although view overlap can occur, it still provides a negative gain for the
right view probability estimation. Disabling $\boldsymbol{f}_{cv}^{(s)}$ also
slightly degrades the performance.

As can be observed both in Table III and in Fig. 5, all models trained through
self-supervision widely outperform the ones trained through supervision. For
both self-supervised and supervised models, a higher maximum disparity
significantly results in a more accurate disparity map, an improved
reconstructed right view and a lower bpsp. However, blindly increasing the
maximum disparity also has a boundary effect, which may even lead to a worse
model.

The common settings for the disparity estimation task are 192 maximum
disparity and supervision. For a fair comparison, some recent disparity
estimation results are compared with our “S-192” L3C-Stereo the Table IV. Our
proposed method, as expected, does not perform very well. Aiming at a better
compression, the encoder modules in L3C-Stereo prefer to extract feature maps
that are easier to compress. Furthermore, the feature maps used for disparity
estimation are all derived from the auxiliary feature maps that have been
squeezed along the channel, which makes it harder for the model to predict an
accurate disparity map. However, our model can still estimate a good disparity
map, which we think is partly due to the use of a multi-scale structure
similar to PSMNet.

One may notice that self-supervised disparity maps are worse than supervised
ones, which is reasonable, since supervised methods require ground-truth,
which contributes to the strong effect of supervision. On the other hand, the
reconstructed right view under self-supervision is better than the one under
supervision. We think that there are some occlusions in the left view, as it
was impossible to reconstruct these areas using the information from the
disparity map and the left view. Expecting a better reconstruction quality,
self-supervision is not attempting to estimate the best disparity map;
instead, it searches for the position in the window with the maximum disparity
length that is most likely to be the pixel value of the right view. In Fig. 5,
compared with the right view, there is significant ghosting in the
reconstructed GT. In fact, these ghostly objects on the right side appear in
the left view image. Since these parts of the area are occluded, they cannot
be reconstructed. As a consequence, the reconstructed GT shows a low PSNR and
SSIM. For instance, in case two, SS (self-supervision) reconstructed right
view results show a shadow near the ball, and the colour of the shadow is
brighter here. Due to occlusion, the left shadow is the information related to
reconstructing the ball in the right view, and the right shadow is the
information related to reconstructing the occluded area. In the reconstructed
right view of “SS-128” and “SS-192”, it can be seen that the results are
relatively better than “S-$n$” models in the ball area.

Figure 5: Case study. Two cases from the testing set are shown, in which one
is easily compressed, and the other is difficult to compress. “S” means
“Supervised” and “SS” means “Self-supervised”. The $n$ in “SS-$n$” and “S-$n$”
refers to the maximum disparity. “GT” means ground-truth. The numbers in the
brackets represent the content in brackets under the left-hand side image of
the same kind. The reconstructed GT is the result of warping the left view
based on the GT disparity map.

###  IV-D Complexity

Table V gives the model parameters, FLOPs and runtime for a pair of $512\times
512$ images (CPU: Intel i5 6500, GPU: Nvidia GTX 980Ti). Compared with L3C,
our method does not significantly increase the computational or time
complexities. For the different maximum disparity, it does not bring in a lot
of FLOPs, and instead brings a greater time consumption. We think that the
warping module does not use the GPU continuously; some operators, such as
generating cost volume and warping images, use a lot of CPU, which could be
optimized.

TABLE V: Complexity Comparison. Model | Encoding | Decoding
---|---|---
Param(M) | FLOPs(G) | Runtime(s) | Param(M) | FLOPs(G) | Runtime(s)
L3C | 4.933 | 352.275 | 0.725 | 2.741 | 226.594 | 0.665
L3C-Stereo(64) | 5.935 | 421.100 | 0.850 | 3.743 | 295.419 | 0.788
L3C-Stereo(128) | 437.548 | 0.937 | 311.868 | 0.871
L3C-Stereo(192) | 453.997 | 1.026 | 328.316 | 0.957

##  V Conclusion

In this paper, we propose the L3C-Stereo method for the compression of stereo
images. By leveraging the disparity map, our method outperforms the compared
methods on three datasets. In the experiments, we show that a significant
margin can be achieved by using the reconstructed right view instead of the
left view. Furthermore, the increase of the maximum disparity hyper-parameter
will lead to a decrease in the running speed of the model and a better
compression effect. However, blindly increasing this parameter will also lead
to a decrease in the model performance. What’s more, the by-products of the
model, the disparity map, still is of a relatively good quality. Future work
should investigate the reconstruction under occlusion, which should use the
texture features near occlusion, and a more efficient way to warp the left
view.

##  VI Acknowledgement

This research was supported by the National Key R&D Program of China (No.
2017YFE0129700), the National Natural Science Foundation of China (Key
Program) (No. 11932013), the National Natural Science Foundation of China (No.
62006045), the Tianjin Natural Science Foundation for Distinguished Young
Scholars (No. 18JCJQJC46100), Japan Society for the Promotion of Science (
Grant-in-Aid for Early-Career Scientists 20K19875).

## References

  * [1] S. Andriani, G. Calvagno, T. Erseghe, G. A. Mian, M. Durigon, R. Rinaldo, M. Knee, P. Walland, and M. Koppetz, “Comparison of lossy to lossless compression techniques for digital cinema,” in 2004 International Conference on Image Processing, 2004. ICIP’04., vol. 1, pp. 513–516, IEEE, 2004\.
  * [2] “libpng home page.” http://libpng.org/pub/png/libpng.html.
  * [3] “Webp image format.” https://developers.google.com/speed/webp.
  * [4] J. Sneyers and P. Wuille, “Flif: Free lossless image format based on maniac compression,” in 2016 IEEE International Conference on Image Processing (ICIP), pp. 66–70, IEEE, 2016.
  * [5] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. V. Gool, “Practical full resolution learned lossless image compression,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10629–10638, 2019.
  * [6] S. Cao, C.-Y. Wu, and P. Krähenbühl, “Lossless image compression through super-resolution,” arXiv preprint arXiv:2004.02872, 2020.
  * [7] F. Mentzer, L. V. Gool, and M. Tschannen, “Learning better lossless compression using lossy compression,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6638–6647, 2020.
  * [8] E. Hoogeboom, J. W. Peters, R. v. d. Berg, and M. Welling, “Integer discrete flows and lossless compression,” NeurIPS, 2019.
  * [9] R. van den Berg, A. A. Gritsenko, M. Dehghani, C. Kaae Sønderby, and T. Salimans, “Idf++: Analyzing and improving integer discrete flows for lossless compression,” arXiv e-prints, pp. arXiv–2006, 2020.
  * [10] H. Zhang, F. Cricri, H. R. Tavakoli, N. Zou, E. Aksu, and M. M. Hannuksela, “Lossless image compression using a multi-scale progressive statistical model,” in Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [11] A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry, “End-to-end learning of geometry and context for deep stereo regression,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 66–75, 2017.
  * [12] J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5418, 2018.
  * [13] G. Yang, H. Zhao, J. Shi, Z. Deng, and J. Jia, “Segstereo: Exploiting semantic information for disparity estimation,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 636–651, 2018.
  * [14] S. Duggal, S. Wang, W.-C. Ma, R. Hu, and R. Urtasun, “Deeppruner: Learning efficient stereo matching via differentiable patchmatch,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4384–4393, 2019.
  * [15] X. Guo, K. Yang, W. Yang, X. Wang, and H. Li, “Group-wise correlation stereo network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3273–3282, 2019.
  * [16] J. Liu, S. Wang, and R. Urtasun, “Dsic: Deep stereo image compression,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3136–3145, 2019.
  * [17] Y. Hu, W. Yang, Z. Ma, and J. Liu, “Learning end-to-end lossy image compression: A benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [18] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” arXiv preprint arXiv:1611.01704, 2016.
  * [19] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” arXiv preprint arXiv:1802.01436, 2018.
  * [20] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” arXiv preprint arXiv:1809.02736, 2018.
  * [21] T. Chen, H. Liu, Z. Ma, Q. Shen, X. Cao, and Y. Wang, “End-to-end learnt image compression via non-local attention optimization and improved context modeling,” IEEE Transactions on Image Processing, vol. 30, pp. 3179–3191, 2021.
  * [22] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. J. Hwang, J. Shor, and G. Toderici, “Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4385–4393, 2018.
  * [23] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306–5314, 2017.
  * [24] C. Lin, J. Yao, F. Chen, and L. Wang, “A spatial rnn codec for end-to-end image compression,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13269–13277, 2020.
  * [25] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, “Conditional image generation with pixelcnn decoders,” arXiv preprint arXiv:1606.05328, 2016.
  * [26] A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent neural networks,” in International Conference on Machine Learning, pp. 1747–1756, PMLR, 2016.
  * [27] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma, “Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications,” arXiv preprint arXiv:1701.05517, 2017.
  * [28] F. Mentzer, Neural Image Compression: Lossy and Lossless Algorithms.  PhD thesis, ETH Zurich, 2021.
  * [29] Y. Bai, X. Liu, W. Zuo, Y. Wang, and X. Ji, “Learning scalable $l_{\infty}$-constrained near-lossless image compression via joint lossy image and residual compression,” arXiv preprint arXiv:2103.17015, 2021.
  * [30] N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4040–4048, 2016.
  * [31] E. Ilg, T. Saikia, M. Keuper, and T. Brox, “Occlusions, motion and depth boundaries with a generic network for disparity, optical flow or scene flow estimation,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 614–630, 2018.
  * [32] Z. Liang, Y. Feng, Y. Guo, H. Liu, W. Chen, L. Qiao, L. Zhou, and J. Zhang, “Learning for disparity estimation through feature constancy,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2811–2820, 2018.
  * [33] J. Pang, W. Sun, J. S. Ren, C. Yang, and Q. Yan, “Cascade residual learning: A two-stage convolutional neural network for stereo matching,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 887–895, 2017.
  * [34] Y. Zhong, Y. Dai, and H. Li, “Self-supervised learning for stereo matching with self-improving ability,” arXiv preprint arXiv:1709.00930, 2017.
  * [35] Z. Jie, P. Wang, Y. Ling, B. Zhao, Y. Wei, J. Feng, and W. Liu, “Left-right comparative recurrent model for stereo matching,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3838–3846, 2018\.
  * [36] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool, “Conditional probability models for deep image compression,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4394–4402, 2018.
  * [37] M. Menze and A. Geiger, “Object scene flow for autonomous vehicles,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3061–3070, 2015.
  * [38] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [39] T. Tieleman and G. Hinton, “Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning,” Technical Report., 2017.