# Accuracy Enhancement for Ear Acoustic Authentication Using Between-class
Features

Masaki YASUHARA
Department of Science of Technology Innovation
Nagaoka University of Technology 1603-1,
Kamitomioka Nagaoka,
Niigata 940-2188, Japan
<EMAIL_ADDRESS>
& Isao Nambu
Graduate School of Engineering,
Nagaoka University of Technology 1603-1,
Kamitomioka Nagaoka,
Niigata 940-2188, Japan
&Yoshiko MARUYAMA
National Institute of Technology,
Hakodate College
14-1 Tokuracho, Hakodate,
Hokkaido, 042-0953, Japan
&Shohei YANO
Department of Electrical and Electronic Systems Engineering,
National Institute of Technology, Nagaoka College
888 Nishikatakai Nagaoka
Niigata 940-8532, Japan

###### Abstract

In existing biometric authentication methods, the user must perform an
authentication operation such as placing a finger in a scanner or facing a
camera. With ear acoustic authentication, acoustic characteristics of the ear
canal are used as biometric information. Therefore, a person wearing earphones
does not need to perform any authentication operation. In biometric
authentication, it is necessary to minimize the false acceptance rate (FAR) so
that no unauthorized user is misidentified as an authorized user. However, if
the FAR is set low, it increases the false rejection rate (FRR), the rate at
which authorized users are falsely recognized as unauthorized users. It has
been reported that when FAR is 0.1%, the FRR in ear acoustic authentication
reaches as much as 22%. In this study, we propose a method that reduces FRR
and enhances authentication accuracy; it generates new ear canal acoustic
characteristics called between-class (BC) features, which combine the ear
canal acoustic characteristics of authorized and unauthorized users features.
The proposed method uses a support vector machine to learn the BC features as
the data of authorized users, then generates a hyperplane in an area close to
that data. We hypothesize that this would decrease the possibility of
misidentifying an unauthorized user as an authorized user and decrease the FRR
when the FAR is 0.1%. To evaluate the performance of the proposed method, BC
features were applied to ear acoustic authentication, and FAR and FRR were
calculated. The FRR with FAR = 0.1% was 7.95% lower than with the previous
method, and the equal error rate – the error rate when FAR and FRR are
equivalent – decreased by 0.15%. These results confirmed that the proposed
method can make ear acoustic authentication more convenient while maintaining
a high level of security. This paper is under peer-previewing of IEICE Trans.
Inf. & Syst.

_Keywords_ Biometrics $\cdot$ Data Augmentation $\cdot$ ECIR $\cdot$ Machine
learning $\cdot$ SVM

##  1 Introduction

Biometric authentication can minimize the risks of information loss, theft, or
leak. Over the past few decades, many types of biometric authentication have
been studied, including recognition by fingerprint, face, iris, retina, and
voice Mizoguchi and Hara (2010); Imaoka (2014); Koshinaka et al. (2015). These
methods require an authentication operation from the user, such as placing a
finger on a scanner or facing a camera. We focused on the ear canal as new
biometric information representing human individuality. Using the ear canal
for identification enables a type of biometric authentication that does not
require any authentication operation; all it requires is to wear an earphone-
shaped authentication device. In the field of sound image localization
technology – which virtually creates the direction of arrival of sound for
earphones and headphones – previous studies have indicated that individual
differences in the outer shape of the ear and the shape of the ear canal
manifest as differences in how people hear sounds Henrik (1992); Wenzel et al.
(1993a); Jens (1997). One of the studies on methods of personal identification
using the ear canal examined the possibility of identifying individuals by
emitting acoustic signals towards the ears and measuring the acoustic
characteristics reflected by the shape of the pinna and the ear canal
Akkermans et al. (2005). However, there were issues with the accuracy of the
measurement of acoustic characteristics and extraction of feature amount that
could be effectively used for authentication. As a result, the expected level
of authentication accuracy was not reached, and this method has not yet been
put into practice.

Using an impulse response measurement method that uses pseudo-noise signals
Borish and Angell (1983), we previously demonstrated a method of personal
authentication that captures sounds reflected in the ear canal with high
accuracy and uses them as features for personal identification and confirmed
the effectiveness of this method through an experiment that employed subjects
Yano et al. (2015). The type of biometric authentication that uses this method
Yano et al. (2015), is called ear acoustic authentication in this paper.
Various studies have sought ways to enhance the accuracy of biometric
authentication using acoustic characteristics of the ear canal. One of these
studies demonstrated that by minimizing measurement errors due to background
noise and producing a constant signal-to-noise ratio (SNR) across all
frequency bands, it is possible to capture ear acoustic authentication
features very accurately Yano et al. (2017). Other studies showed the
effectiveness of a method that reduces observation fluctuations by additive
averaging Moriya and Kaneda (2008); Yano et al. (2017). A method of ear
acoustic authentication using inaudible sounds was examined by Mahto et al.,
who built a hybrid system that uses both audible and inaudible areas Mahto et
al. (2018). These ear acoustic authentication methods used the features of
only one ear; we demonstrated that combining features of both ears increases
authentication accuracy Yasuhara et al. (2019).

Misidentification of an unauthorized user as an authorized user is a major
issue in biometrics authentication. In ear acoustic authentication, when the
false acceptance rate (FAR)– the probability of incorrectly accepting
unauthorized users – is 0.1%, the false rejected rate (FRR) – the probability
of incorrectly rejecting an authorized user – is 22% Yasuhara et al. (2019).
That is, if a biometric authentication system is operated at 0.1% FAR, it
incorrectly recognizes an authorized user as an unauthorized user once every
five times. Therefore, it is necessary to enhance the accuracy of ear acoustic
authentication to make it more convenient while maintaining a high security
level. In this study, we propose a method that improves authentication
accuracy by generating between-class (BC) features that combine the ear
acoustic features of users. In this study, we generate BC features for a
dataset of 52 subjects, carry out 1:1 (one-to-one) authentication using a
support vector machine (SVM) and empirically evaluate its effectiveness.

##  2 Ear Acoustic Authentication System

In ear acoustic authentication, biometric authentication is performed using a
special earphone with a small microphone embedded, as illustrated in Fig. 1.
It emits a maximum-length sequence (MLS) signal from the speaker inside the
earphone and the microphone records the sound echoed from the ear canal to
capture its acoustic characteristics. These ear canal acoustic characteristics
include reflection, diffraction, interference, and resonance characteristics,
which vary from person to person according to the shape of the ear canal and
the body. To perform biometric authentication, ear acoustic authentication
focuses on the individuality of these ear canal acoustic characteristics
Wenzel et al. (1993b); Yano et al. (2000), which go through feature amount
extraction and are applied to a discriminator.

In this study, we assume the ear acoustic authentication system illustrated in
Fig. 2. This system runs with 1:1 authentication, a type of authentication
that initially identifies the user, then identifies whether the measured
biometric information belongs to that user. As the discriminator of
authentication, we use a support vector machine (SVM), a type of supervised
learning used for classification, regression, and outlier detection. It can
also calculate posterior probabilities as suggested by Platt al. Platt (1999)
and Wu al. Wu et al. (2004). If the posterior probability is high, the
probability that it is the identified user is high. Ultimately, it compares
the posterior probability to a threshold and outputs “Accept” or “Reject”.

Figure 1: A special earphone which has a speaker and a microphone Figure 2:
Overview of ear acoustic authentication system

###  2.1 Factors of Evaluation

In 1:1 authentication, the biometric information provided is classified as
belonging to an authorized user or not. As mentioned before, FAR is the rate
at which biometric information belonging to an unauthorized user is
incorrectly accepted as that of an authorized user, and FRR is the rate at
which biometric information belonging to an authorized user is incorrectly
rejected as that of an unauthorized user. True acceptance rate (TAR) is the
rate at which biometric information of an authorized user is correctly
accepted as that of an authorized user. These are defined by the following
equations:

$\displaystyle FAR$ $\displaystyle=\frac{T_{FA}}{N_{auth}}$ (1) $\displaystyle
FRR$ $\displaystyle=\frac{T_{FR}}{N_{auth}}$ (2) $\displaystyle TAR$
$\displaystyle=1-FRR$ (3)

Here, $N_{auth}$ is the number of authentications performed, $T_{FR}$ is the
number of authorized users incorrectly rejected, and $T_{FA}$ is the number of
unauthorized users incorrectly accepted. When doing the classification, it is
necessary to set a threshold that determines the minimum probability of being
an authorized user to be accepted. If this threshold is set low, it is more
difficult to incorrectly reject an authorized user and easier to incorrectly
accept an unauthorized user. On the contrary, if the threshold is set high, it
becomes easier to reject authorized users. Therefore, as indicated in Fig.
3(a), there is a trade-off between FAR and FRR. The graph shown in Fig. 3(a),
indicating the relationship between FAR and FRR, is known as a detection error
tradeoff (DET) curve. The intersection between FAR and FRR is called the equal
error rate (EER), which is used as a performance indicator of biometric
authentication. The lower the EER, the higher the authentication accuracy. The
graph shown in Fig. 3(b), indicating the relationship between TAR and FAR, is
known as a receiver operating characteristic (ROC) curve. The area below the
ROC curve is called area under the curve (AUC) and it is also used as a
performance indicator of biometric authentication. AUC varies from 0 to 1; if
there were no authentication mistakes, AUC is 1, and in the case of accidental
authentication, 0.5. Therefore, the higher the AUC, the higher the
authentication accuracy.

Figure 3: DET curve and ROC curve. (a) DET curve : FAR and FRR against
threshold. (b) ROC curve : TAR against FAR.

##  3 Between-class feature and related works

BC features is the feature amount combining the training samples of authorized
users, $x_{1}$, and training samples of unauthorized users, $x_{2}$, with
ratio $r(0<r<1)$ as per the following equation:

$rx_{1}+(1-r)x_{2}$ (4)

Here, $\bf{X_{1}}$ is the group of training samples of authorized users,
$\bf{X_{2}}$ is the group of training samples of unauthorized users, so
$x_{1}\in\bf{X_{1}}$ and $x_{2}\in\bf{X_{2}}$. We considered the possibility
of learning the BC features as unauthorized users when performing 1:1
authentication by SVM and thereby reducing the FAR of ear acoustic
authentication. The expected effect of performing 1:1 authentication on two-
dimensional data of authorized and unauthorized users using an SVM can be
explained as follows. In previous ear acoustic authentication, a hyperplane
that maximizes the margin of each feature amount, in the position indicated by
the dotted line in Fig. 4, was chosen. However, when generating BC features
and treating them as unauthorized users, the hyperplane is expected to move
closer to the data of authorized users. With this, it becomes more difficult
to incorrectly accept unauthorized users, and if the threshold of the
biometric authentication system is set low, fewer unauthorized users would be
incorrectly accepted.

Therefore, the FAR curve is expected to drop rapidly as the threshold
increases, as illustrated in Fig. 5. On the other hand, when the hyperplane
gets closer to the data of authorized users, the probability of incorrectly
rejecting authorized users could increase. In other words, the steepness of
the entire FRR curve is expected to increase. However, how the FAR and FRR
curves change because of BC features is unknown. The ratio $r$ and $N_{BC}$
(the number of BC features) are parameters thought to be closely related to
authentication accuracy. Since BC features generate features in the middle of
two classes, ideally, the difference between the features ($x_{1}$ and$x_{2}$)
that will generate the BC features should be small. Therefore, the proposed
method calculates the cosine similarity between $x_{1}$ and $x_{2}$ and
generates $N_{BC}$ units of BC features from the combination with high cosine
similarity, as in Eq. 5:

$\cos(x_{1},x_{2})=\frac{x_{1}\cdot x_{2}}{|x_{1}||x_{2}|}$ (5)

Figure 4: Hyperplane in feature space that changes with the introduction of BC
features Figure 5: The lower FAR and higher FRR with introducing of BC
features compared to previous method which single ear data without BC

Other studies have sought to improve the classification accuracy by combining
two training samples. Mix-up Zhang et al. (2017) is a data augmentation method
that mixes pairs of two training samples to generate new training samples.
Between-class learning Tokozume et al. (2018) is a data augmentation method
that mixes training sample pairs of two different classes to generate new
training samples. The ratio $r$ used to combine two feature amounts of mix-up
is a random number that follows a beta distribution, and in between-class
learning, it is a random number that follows a uniform distribution. Mix-up
and between-class learning are data augmentation methods in deep learning. In
these methods, training samples combining two features with random proportions
are treated as new classes. This smooths the decision boundaries and makes
over-learning difficult to occur. Meanwhile, the proposed method aims to bring
the separation interface in 1:1 authentication closer to the data of
authorized users. By learning training samples that combine two features with
a constant rate as unauthorized users, this method brings the separation
interface closer to the data of authorized users to reduce the FAR. Therefore,
the related studies and the proposed method have different objectives,
particularly the fact that the ratio $r$ used to combine two features is a
constant in the proposed method.

##  4 Experimental methods

To evaluate the effect of introducing BC features, 1:1 authentication was
performed using the dataset of ear acoustic authentication used in a previous
study Yasuhara et al. (2019). It was then evaluated by calculating the FAR,
FRR, EER, and AUC.

###  4.1 Dataset

The data used in this experiment are the same data used in a previous study
Yasuhara et al. (2019). We enrolled 52 students and employees of the National
Institute of Technology, Nagaoka College aged between late teens and 50s.
Following the college rules, we obtained informed consent from all subjects to
participate in this study. The flow of the measurement of ear canal acoustic
characteristics and preprocessing is shown in Fig. 6. The ear canal acoustic
characteristics of each person were measured 30 times using a canal-type
earphone (Bose SoundTrue Ultra). An MLS signal $m[n]$ (signal length
$n=2^{14}-1$) was emitted from the earphone with a noise level of 65 dB(A) and
its reflected sound $y[n]$ was measured with a small microphone. This
measurement was done five times and the averaged signal $\bar{y}[n]$ was
subject to Hadamard transform to derive the impulse response of acoustic
transmission characteristics $h[n]$ of the ear canal. The feature amount of
both ears was captured, but since only one ear was enough to check the
effectiveness of BC features, we used the data from the left ear of all
subjects.

Figure 6: The flow of data measurement and preprocessing

###  4.2 Preprocessing

When minimum-phase transformation using Hilbert transform is applied to
$h[n]$, the impulse response components comprising the features were found in
the head of the signal. After normalizing the power of all signals to 1 and
clipping 256 samples from the head, bandpass filtering (100 Hz–22 kHz) was
performed. This is used as the feature amount $f[n]$.

###  4.3 Classification

As mentioned before, this study is based on 1:1 authentication, which
identifies whether the feature amount input belongs to an authorized user or
an unauthorized user. The classification is performed using the SVC function,
an SVM for classification implemented in scikit-learn Pedregosa et al. (2011).
The SVC function was implemented based on libsvm Chang and Lin (2011) and can
specify the kernel and regularization parameters used. The data of authorized
users were divided into six samples of training data and 24 samples of test
data. For the data of unauthorized users, we excluded the data of one person
for test data, and the rest data for learning. BC features were generated from
the training data of authorized users and unauthorized users and were treated
as training data of unauthorized users.

In the example of Fig. 7, Person 1 is the authorized user, Person 2 is the
unauthorized user for test data, and Person 3 to Person 52 are the
unauthorized users for learning. The authorized user and the unauthorized user
for test form ${}_{52}\mathrm{P}_{2}=2,652$ possible combinations, and the SVM
learned each combination and calculated the accuracy. For each sample of test
data, the SVC function calculates the probability of being an authorized user
in the range from 0 to 1. The FAR, FRR, and TAR when the threshold is changed
are calculated from the estimated probability using the roc_curve function of
scikit-learn. To calculate the posterior probability, the probability
parameter of the SVC function was set to True, and the other parameters of the
SVC function were set to default.

The ratio of BC features, $r$, was changed to 11 different values: 0.01 and
0.05, then 0.1 to 0.9 with an increment of 0.1. The number of BC features,
$N_{BC}$, was changed to 9, 45, 90, 450, 900, 4,500, and 9,000. As an example,
in Fig. 7, when $N_{BC}$ is 9,000, there are 6 data samples of an authorized
user for learning, 1,500 data samples of unauthorized users for learning, and
9,000 data samples generated by BC features for learning, totaling 10,506 data
samples for learning.

Figure 7: Training and test dataset. Person 1 is the authorized user, Person 2
is unauthorized user for test data, and Person 3 through Person 52 are the
unauthorized users for learning.

##  5 Result

Table 1 shows the conditions that produced more accurate results than the
previous method in all five evaluation indexes. The conditions that produced
the highest accuracy were $r=0.4$ and $N_{BC}=4,500$, generating an EER of
1.56%, which is 0.15% lower than the previous method which produces single ear
data without BC features Yasuhara et al. (2019). The FRR with FAR = 0.1% was
14.35%, or 7.95% lower than the previous method. Fig. 8(a) shows the DET curve
with $r=0.4$ and $N_{BC}=4,500$, and Fig. 8(b) is an enlarged view of part of
Fig. 8(a). As a result of learning the BC features as unauthorized users, FAR
dropped sharply, and FRR increased as the threshold increased. EER moved
toward lower thresholds as the FAR and FRR curves changed.

The results with all conditions are shown in Table 2. The numbers in bold are
those that were more accurate than the previous method. The stars in the table
indicate conditions that produced more accurate results than the previous
method in all five evaluation indexes.

Figure 8: DET curves of $r=0.4,N_{BC}=4500$ condition. (a) overall view (b)
enlarged view

##  6 Discussion

###  6.1 Validity of accuracy enhancement by BC features

Since BC features generate intermediate features of authorized and
unauthorized users, they may have a negative impact on the learning of
discriminators such as SVM. Because of the trade-off between FAR and FRR, even
if the introduction of BC features results in a lower FAR, it cannot be
considered more accurate if EER increases or AUC decreases. In this
experiment, we introduced BC features using parameters ($r$ and $N_{BC}$)
found by grid search, and as a result, four sets of conditions produced more
accurate results (all evaluation indexes) than the previous method. These
results indicate that by setting appropriate BC features parameters, $r$ and
$N_{BC}$, we were able to enhance the authentication accuracy by decreasing
not only FAR but also EER without impacting the learning of the discriminator
(SVM in this case).

###  6.2 Comparison between the expected DET curve and the DET curve obtained
from the experiment

A comparison between the expected DET curve, shown in Fig. 5, and the DET
curve obtained from the experiment, in Fig. 8, shows that in both graphs, FAR
rapidly decreased and FRR increased as the threshold increased. EER also moved
toward smaller thresholds with the changes in the FAR and FRR curves. Based on
this, it can be said that the proposed method successfully reduced FAR by
learning BC features as unauthorized users.

###  6.3 Changes in DET curves due to differences in BC feature parameters

Fig. 9 shows DET curves when $N_{BC}$ is 90, 4,500, and 9,000. The FAR curves
when BC features are introduced show no significant difference between an
$N_{BC}$ of 4,500 and 9,000. On the other hand, the FRR curves show bigger
differences compared to the previous method as $N_{BC}$ increases. We expected
that as $N_{BC}$ was increased, FAR would continue to decrease and FRR would
continue to increase, but in practice, FAR did not continue to decrease when
$N_{BC}$ was increased. With an $N_{BC}$ of 4,500 and 9,000, FAR decreased
significantly, but there was no big difference between these two conditions.
On the other hand, FRR was higher when $N_{BC}$ was 9,000 than when it was
4,500, which makes 4,500 a better parameter. In this experiment, all four
conditions that improved all accuracy evaluation indexes had an $N_{BC}$ of
4,500. Based on this, it is possible to conclude that the $N_{BC}$ of 4,500 is
the ideal parameter that can reduce FAR with a minimal increase in FRR.

As for $r$, the FAR when $N_{BC}$ is 90 and 4,500 barely changes until $r$
reaches 0.4. On the other hand, the FRR graph when $N_{BC}$ is 4,500 shows
that the larger the $r$, the lower the FAR. In this experiment, the four
conditions that enhanced all accuracy evaluation indexes had an $r$ of 0.1,
0.2, 0.3, or 0.4. Therefore, an $N_{BC}$ of 4,500 and an $r$ between 0.1 and
0.4 were the right parameters that reduced FAR without increasing FRR.

Figure 9: FAR and FRR curves with various parameters. (a) FAR curve of $N_{BC}=90$. (b) FRR curve of $N_{BC}=90$. (c) FAR curve of $N_{BC}=4500$. (d) FRR curve of $N_{BC}=4500$. (e) FAR curve of $N_{BC}=9000$. (f) FRR curve of $N_{BC}=9000$. Table 1: Classification performance of conditions that produced more accurate results than the previous method in all five evaluation indexes. $r$ | $N_{BC}$ | AUC | EER (%) | FRR at FAR=0.01 (%) | FRR at FAR=0.1 (%) | FRR at FAR=1 (%)
---|---|---|---|---|---|---
Previous method | 0.9983 | 1.71 | 78.05 | 22.30 | 2.45
0.1 | 4500 | 0.9984 | 1.62 | 69.53 | 20.01 | 2.36
0.2 | 4500 | 0.9985 | 1.61 | 56.23 | 17.93 | 2.15
0.3 | 4500 | 0.9984 | 1.56 | 56.10 | 16.04 | 2.20
0.4 | 4500 | 0.9985 | 1.61 | 56.23 | 17.93 | 2.15

##  7 Conclusions

One problem reported in ear acoustic authentication, which is a type of
biometric authentication, is that it yields a high FRR when FAR = 0.1%; that
is, when the security level is increased, the ear acoustic authentication
becomes less convenient. Therefore, in this study, we proposed a method to
enhance the accuracy of ear acoustic authentication systems by reducing the
FRR when FAR = 0.1% based on BC features. BC features are intermediate
features of authorized and unauthorized users. We hypothesized that if a
discriminator (SVM) learned BC features as unauthorized users, the number of
unauthorized users incorrectly recognized as authorized users would decrease;
and with a lower FAR, the FRR when FAR = 0.1% would also decrease. The results
were as hypothesized; the use of BC features decreased the FRR by 7.95% when
FAR = 0.1%, with the EER being reduced by 0.15%, and the acoustic
authentication accuracy improved. Additionally, a grid search was required to
obtain BC feature parameters suitable for the dataset when generating BC
features. In this study, we devised BC features to reduce the FRR of ear
acoustic authentication, and achieved decreases in the FRR and improvements in
the classification accuracy. These BC features may be applicable to other
biometrics where there are trade-offs between FAR and FRR; thus, further
studies need to be conducted.

Table 2: Classification performance with varying $r$ and $N_{BC}$. The star *
indicates the performance is beyond previous method by all evaluation factors.

$r$ | $N_{BC}$ | AUC | EER (%) | FRR at FAR=0.01 (%) | FRR at FAR=0.1 (%) | FRR at FAR=1 (%)
---|---|---|---|---|---|---
Previous method | 0.9983 | 1.71 | 78.05 | 22.30 | 2.45
| 9 | 0.9983 | 1.68 | 78.84 | 22.10 | 2.46
| 45 | 0.9983 | 1.70 | 75.93 | 21.83 | 2.46
| 90 | 0.9983 | 1.67 | 74.74 | 22.31 | 2.40
0.01 | 450 | 0.9983 | 1.66 | 78.20 | 22.40 | 2.40
| 900 | 0.9983 | 1.65 | 76.57 | 23.50 | 2.40
| 4500 | 0.9983 | 1.62 | 72.68 | 22.10 | 2.51
| 9000 | 0.9983 | 1.73 | 82.35 | 23.83 | 2.74
| 9 | 0.9983 | 1.68 | 78.66 | 22.10 | 2.46
| 45 | 0.9983 | 1.70 | 75.91 | 22.33 | 2.46
| 90 | 0.9983 | 1.66 | 76.07 | 22.28 | 2.40
0.05 | 450 | 0.9983 | 1.66 | 76.16 | 22.15 | 2.40
| 900 | 0.9983 | 1.65 | 74.14 | 23.25 | 2.40
| 4500 | 0.9984 | 1.61 | 69.66 | 21.27 | 2.50
| 9000 | 0.9983 | 1.73 | 82.78 | 23.00 | 2.69
| 9 | 0.9983 | 1.68 | 78.81 | 22.06 | 2.46
| 45 | 0.9983 | 1.70 | 76.40 | 22.20 | 2.46
| 90 | 0.9983 | 1.66 | 75.12 | 22.27 | 2.40
0.1 | 450 | 0.9983 | 1.66 | 73.16 | 22.07 | 2.46
| 900 | 0.9983 | 1.65 | 71.09 | 22.71 | 2.39
| *4500 | 0.9984 | 1.62 | 69.53 | 20.01 | 2.36
| 9000 | 0.9983 | 1.72 | 85.19 | 21.90 | 2.65
| 9 | 0.9983 | 1.68 | 78.63 | 22.07 | 2.46
| 45 | 0.9983 | 1.70 | 76.17 | 22.60 | 2.46
| 90 | 0.9983 | 1.66 | 76.40 | 22.21 | 2.40
0.2 | 450 | 0.9982 | 1.64 | 70.48 | 21.67 | 2.47
| 900 | 0.9983 | 1.66 | 69.17 | 21.43 | 2.41
| *4500 | 0.9985 | 1.61 | 56.23 | 17.93 | 2.15
| 9000 | 0.9984 | 1.71 | 88.47 | 20.88 | 2.22
| 9 | 0.9983 | 1.69 | 78.98 | 22.13 | 2.46
| 45 | 0.9983 | 1.70 | 78.53 | 22.48 | 2.46
| 90 | 0.9983 | 1.71 | 77.47 | 21.62 | 2.40
0.3 | 450 | 0.9981 | 1.68 | 72.64 | 19.47 | 2.40
| 900 | 0.9982 | 1.69 | 68.60 | 18.87 | 2.40
| *4500 | 0.9984 | 1.56 | 56.10 | 16.04 | 2.20
| 9000 | 0.9983 | 1.67 | 90.59 | 21.12 | 2.12
| 9 | 0.9983 | 1.70 | 79.56 | 21.90 | 2.46
| 45 | 0.9983 | 1.73 | 79.38 | 22.01 | 2.48
| 90 | 0.9984 | 1.72 | 78.23 | 21.10 | 2.53
0.4 | 450 | 0.9980 | 1.68 | 66.61 | 19.16 | 2.42
| 900 | 0.9981 | 1.68 | 67.38 | 19.31 | 2.40
| *4500 | 0.9984 | 1.56 | 56.68 | 14.35 | 2.29
| 9000 | 0.9983 | 1.59 | 95.73 | 23.92 | 2.12
| 9 | 0.9983 | 1.68 | 80.51 | 21.85 | 2.41
| 45 | 0.9983 | 1.67 | 79.49 | 21.88 | 2.55
| 90 | 0.9984 | 1.70 | 78.14 | 21.36 | 2.57
0.5 | 450 | 0.9980 | 1.67 | 64.43 | 19.68 | 2.48
| 900 | 0.9981 | 1.66 | 64.55 | 19.62 | 2.39
| 4500 | 0.9982 | 1.70 | 60.65 | 15.42 | 2.49
| 9000 | 0.9981 | 1.51 | 96.83 | 27.18 | 2.05
| 9 | 0.9984 | 1.69 | 71.40 | 21.40 | 2.63
| 45 | 0.9983 | 1.81 | 73.32 | 23.07 | 2.80
| 90 | 0.9983 | 1.81 | 74.38 | 20.77 | 2.88
0.6 | 450 | 0.9978 | 1.76 | 71.89 | 20.10 | 2.78
| 900 | 0.9980 | 1.73 | 74.38 | 20.44 | 2.75
| 4500 | 0.9979 | 1.70 | 68.22 | 17.43 | 2.65
| 9000 | 0.9979 | 1.44 | 99.52 | 35.83 | 1.89
| 9 | 0.9983 | 1.71 | 76.59 | 21.96 | 2.64
| 45 | 0.9982 | 1.92 | 82.12 | 21.15 | 3.23
| 90 | 0.9981 | 1.93 | 81.78 | 19.76 | 3.25
0.7 | 450 | 0.9976 | 1.90 | 64.34 | 24.26 | 3.20
| 900 | 0.9978 | 1.83 | 64.44 | 21.12 | 3.17
| 4500 | 0.9975 | 1.59 | 81.07 | 26.51 | 2.98
| 9000 | 0.9975 | 1.46 | 99.91 | 60.60 | 1.91
| 9 | 0.9980 | 1.99 | 82.78 | 28.83 | 2.97
| 45 | 0.9979 | 2.05 | 92.40 | 26.80 | 3.42
| 90 | 0.9977 | 2.14 | 89.38 | 29.47 | 3.29
0.8 | 450 | 0.9972 | 2.12 | 85.45 | 28.01 | 3.59
| 900 | 0.9972 | 1.95 | 85.08 | 29.52 | 3.41
| 4500 | 0.9968 | 1.85 | 92.72 | 49.41 | 3.46
| 9000 | 0.9963 | 1.75 | 99.92 | 89.58 | 3.91
| 9 | 0.9959 | 2.86 | 94.62 | 50.48 | 6.77
| 45 | 0.9961 | 2.45 | 99.78 | 59.90 | 6.44
| 90 | 0.9960 | 2.53 | 100.00 | 69.56 | 6.53
0.9 | 450 | 0.9955 | 2.43 | 100.00 | 69.90 | 6.11
| 900 | 0.9952 | 2.17 | 100.00 | 94.92 | 5.65
| 4500 | 0.9953 | 2.05 | 99.34 | 76.87 | 6.70
| 9000 | 0.9887 | 4.70 | 99.77 | 92.23 | 11.87

## Acknowledgments

This work was supported by JSPS KAKENHI Grant Numbers JP19H04112, JP19K22851.

## References

  * Mizoguchi and Hara [2010] Masanori Mizoguchi and Masanori Hara.  Fingerprint/palmprint matching identification technology.  _NEC Technical Journal_ , 5(3):18–22, 2010.
  * Imaoka [2014] Hitoshi Imaoka.  Face recognition research: Beyond the limit of accuracy.  In _The IAPR 2014 Biometrics Lecture, The 2014 International Joint Conference on Biometrics (IJCB 2014)_ , 2014.
  * Koshinaka et al. [2015] Takafumi Koshinaka, Osamu Hoshuyama, Yoshifumi Onishi, and Masahiro Tani.  Speech/acoustic analysis technology - its application in support of public solutions.  _NEC Technical Journal_ , 9(1):82–85, 2015.
  * Henrik [1992] Møller Henrik.  Fundamentals of binaural technology.  _Applied Acoustics_ , 36:171–218, 1992.
  * Wenzel et al. [1993a] Elizabeth M. Wenzel, Arruda Marinne, Doris J. Kistler, and Frederic L. Wightman.  Localization using nonindividualized head-related transfer functions.  _J. Acoust. Soc. Am._ , 94(1):111–123, July 1993a.
  * Jens [1997] Blauert Jens.  _Spatial Hearing_.  MIT Press, 1997.
  * Akkermans et al. [2005] A. H. M. Akkermans, T. A. M. Kecenaar, and D. W. E. Schobben.  Acoustic ear recognition for person identifications.  In _Fourth IEEE Workshop on Automatic Identification Advanced Technologies(AutoID’05)_ , pages 219–223, 2005.
  * Borish and Angell [1983] Jeffrey Borish and James B. Angell.  An efficient algorithm for measuring the impulse response using pseudorandom noise.  _J. Audio Eng. Soc_ , 31(7/8):478–488, 1983.
  * Yano et al. [2015] Shohei Yano, Takayuki Arakawa, Takafumi Koshinaka, and Hitoshi Irisawa.  Peoposal of authentication method by acoustical characterstics of ear canal.  In _The Fifth Symposium on Biometrics, Recognition and Authentication(SBRA)_ , pages 94–95, Nov 2015.
  * Yano et al. [2017] Shouhei Yano, Takayuki Arakawa, Takafumi Koshinaka, Hitoshi Imaoka, and Hideki Irisawa.  Improving acoustic ear recognition accuracy for personal identification by averaging biometric data and spreading measurement errors over a wide frequency range.  _The IEICE Transactions_ , J100-A(4):161–168, 2017.
  * Moriya and Kaneda [2008] Naoya Moriya and Yutaka Kaneda.  Optimum signal for impulse response measurement that minimizes error caused by ambient noise.  _THE JOURNAL OF THE ACOUSTICAL SOCIETY OF JAPAN_ , 64(12):695–701, 2008.
  * Mahto et al. [2018] Shivangi Mahto, Takayuki Arakawa, and Takafumi Koshinaka.  Ear acoustic biometrics using inaudible signals and its application to continuous user authentication.  In _Proc. 28th European Signal Processing Conf._ , 2018.
  * Yasuhara et al. [2019] Masaki Yasuhara, Shohei Yano, Takayuki Arakawa, and Takafumi Koshinaka.  Feature selection and its evaluation in binaural ear acoustic authentication.  In _Audio Engineering Society Convention 146_ , Mar 2019.
  * Wenzel et al. [1993b] Elizabeth M. Wenzel, Marianne Arruda, Doris J. Kistler, and Frederic L. Wightman.  Localization using non-individualized head-related transfer functions.  _J. Acoust. Soc. Am._ , 94(1):111–123, 1993b.
  * Yano et al. [2000] Shohei Yano, Haruhide Hokari, and Shoji Shimada.  A study on the personal difference in the transfer functions of sound localization using stereo earphones.  _IEICE Trans. Fundamentals_ , E83-A(5):877–887, May 2000.
  * Platt [1999] John C. Platt.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  In _ADVANCES IN LARGE MARGIN CLASSIFIERS_ , pages 61–74. MIT Press, 1999.
  * Wu et al. [2004] Ting-Fan Wu, Chih-Jen Lin, and Ruby C. Weng.  Probability estimates for multi-class classification by pairwise coupling.  _J. Mach. Learn. Res._ , 5:975â1005, December 2004.  ISSN 1532-4435.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _CoRR_ , abs/1710.09412, 2017.
  * Tokozume et al. [2018] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada.  Learning from between-class examples for deep sound recognition.  In _International Conference on Learning Representations_ , 2018.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Chang and Lin [2011] Chih-Chung Chang and Chih-Jen Lin.  LIBSVM: A library for support vector machines.  _ACM Transactions on Intelligent Systems and Technology_ , 2:27:1–27:27, 2011.


# StereoISP: Rethinking Image Signal Processing for Dual Camera Systems

Ahmad Bin Rabiah
Purdue University
<EMAIL_ADDRESS>Qi Guo
Purdue University
<EMAIL_ADDRESS>
###### Abstract

Conventional image signal processing (ISP) frameworks are designed to
reconstruct an RGB image from a single raw measurement. As multi-camera
systems become increasingly popular these days, it is worth exploring
improvements in ISP frameworks by incorporating raw measurements from multiple
cameras. This manuscript is an intermediate progress report of a new ISP
framework that is under development, StereoISP. It employs raw measurements
from a stereo camera pair to generate a demosaicked, denoised RGB image by
utilizing disparity estimated between the two views. We investigate StereoISP
by testing the performance on raw image pairs synthesized from stereo
datasets. Our preliminary results show an improvement in the PSNR of the
reconstructed RGB image by at least 2dB on KITTI 2015 and drivingStereo
datasets using ground truth sparse disparity maps.

##  1 Introduction

Multi-camera systems have been unprecedentedly useful on various artificial
platforms such as mobile phones, autonomous vehicles, and drones. For example,
the dual camera system on the latest iPhone 14 can synthesize large aperture
photos and achieve continuous, hybrid-optical-and-digital zooming. Given their
growing popularity, it is worth asking how image signal processing (ISP)
framework can be improved by leveraging raw measurements from multiple
cameras.

In digital photography, the ISP framework transforms a color-mosaicked, noisy
raw measurement $M$ to an RGB image $O$. The conventional ISP framework
sequentially performs demosaicking and denoising on the raw image $M$ (Fig.
1(a)). Since the birth of digital photography, there have been numerous
methods on improving this ISP framework [18, 7, 8, 23, 17, 22, 16]. Some most
recent ones include: switching the sequence of demosaicking and denoising
[22], performing demosaicking and denoising simultaneously [7, 24], and using
burst captures to improve the quality of the final output [16, 17]. But to the
best of our knowledge, there have been very limited studies on designing ISP
frameworks specialized for multi-camera systems, despite their recent
popularity.

(a) Conventional ISP

(b) StereoISP

Figure 1: StereoISP vs. conventional ISP. (a) Conventional image signal
processing (ISP) recovers an RGB image from a single raw measurement by
sequentially performing demosaicking $\mathcal{D_{M}}$ and denoising
$\mathcal{D_{N}}$. (b) StereoISP is specialized for dual camera systems. It
utilizes raw measurements $M$ and $S$ captured by the cameras to output an RGB
image with improved quality. The diagram shows the ultimate framework. It
feeds the raw image pair $M$ and $S$ into a stereo matching module
$\mathcal{G}$ to estimate the disparity between raw images. The warping module
$\mathcal{W}$ warps $S$ according to the estimated disparity to align with
$M$. Finally, the framework sends the raw measurement $M$ and the raw
measurement warped from $S$ into a joint demosaicking and denoising module
$\mathcal{D_{M}D_{N}}$ to generate the RGB image $O$. At the current stage, we
haven’t successfully developed the stereo matching module $\mathcal{G}$, so
our experiment uses ground truth disparity maps between $M$ and $S$.

We propose StereoISP. It is an ISP framework that takes in raw measurements
$M,S$ from a dual camera system and outputs an RGB image $O$ (Fig. 1(b)).
StereoISP utilizes a deep stereo-matching model to find the disparity map
between the raw image pairs. It warps one of the raw measurement according to
the estimated disparity to align with the other, and then feeds both raw
measurements into a deep neural network for joint denoising and demosaicking.

There are two key components that empowers StereoISP. First, the stereo-
matching architecture in StereoISP uses raw stereo pairs as inputs. We
demonstrate experimentally that the accuracy of the output disparity map is
close to that of using RGB image pairs as inputs. Second, StereoISP explicitly
performs alignment of raw image pairs via warping before denoising and
demosaicking. This is different from recent work in burst photography [16, 21]
that directly use unwarped images for restorations. We show via ablation study
that the output image quality of StereoISP drops to the level of single image
ISP if the warping operation is removed, which suggests its importance. This
is probably because the disparity between stereo image pairs are
systematically larger than the disparity among burst images. It is more
challenging to implicitly learn the pixel correspondences between images while
performing denoising and demosaicking when the disparity is large.

At the current stage, we test the performance of StereoISP on raw image pairs
synthesized from stereo datasets, KITTI 2015 [20] and DrivingStereo [25]. Our
current experiment shows that StereoISP boosts the PSNR of the output RGB
image by 2.32dB and 2.43dB on KITTI 2015 and drivingStereo datasets using the
ground truth sparse disparity map. As discussed above, we also provide
comprehensive ablation studies to validate stereo matching on raw images, the
importance of the warping operation for the framework.

The contribution of this papers include:

  * •

A novel ISP framework specialized for dual camera systems

  * •

A comprehensive experimental study to understand the proposed ISP framework

The code and the pre-trained models of StereoISP will be published online for
the community to use and research.

Figure 2: Stereo DemosaicNet inspired by original DemosaicNet [7] that
provides joint demosaicking and denoising for a single camera. Without the
loss of generality of StereoISP, we extended on DemosaicNet to explicitly
align the warped image. Note that the model architecture follows Eq. 1 and
assumed that output of $\mathcal{G}$ has been founded and warped $W$ obtained
from $\mathcal{W}$.

##  2 Related Work

Deep stereo matching. Designing a good structure for stereo matching is
challenging, as it consists of multiple stages that need to be integrated: (1)
feature networks extracting general context from 2D input images. (2) cost
volume computation which stores the distance between 2D features. (3) 3D
matching networks. (4) a powerful disparity regression module to estimate the
continuous disparity map [12]. MC-CNN [26] is the first deep learning-based
stereo matching method. It replaces the first stage of stereo matching (the
matching cost computation) by learning a similarity measure on small image
patches exploiting convolutional neural networks. The first End-to-End
network, namely DispNet, has been developed in [19]. Authors in [2] proposed
PSMNet, a stereo matching model that exploits spatial pyramid pooling module
[10] for effective incorporation of the global context of the stereo pair.
Recently, LEAStereo [3] is proposed which is an end-to-end hierarchical neural
architecture search stereo matching network. the main contribution is the
volumetric stereo-matching pipeline which allows the network to automatically
select the optimal structures for feature extracting and matching networks.

Single image denoising. Single image denoising methods can be categorized into
two categories: model-based and learning-based methods. BM3D [4] falls into a
model-based category, which is regarded as a denoising benchmark. In 2017,
DnCNN [27] was proposed by exploiting deep convolutional nets and adopting
residual learning for blind zero-mean noise denoising. Later on, different
learning-based methods were proposed in different domains, not only in
supervised learning manner [15, 9, 14].

Joint denoising and demosaicking. Demosaicking becomes complicated in the
presence of noise. Practically, the output of image sensors will affect the
quality of reconstructed images. Estimating edge orientation in noisy data is
sensitive which may lead to annoying artifacts in the restored images. The
first deep joint demosaicking and denoising work was done in [7] in which
authors proposed DemosaicNet, a deep CNN-based demosaicking and denoising
algorithm. DemosaicNet is trained on millions of carefully selected image
patches and set a milestone on joint deep learning-based demosaicking and
denoising approaches. In 2019, an extensive study on the previous work was
done in [5]. Authors in [22] proposed a trinity network (TENet) to jointly
solve this composite problem with the super-resolution output. The authors
analyzed the mixture problem of demosaicking, denoising, and super-resolution
and proposed a new pipeline by moving demosaicking process after the denoising
stage. Most recently, authors in [24] proposed a deep learning-based end-to-
end method for joint image demosaicking, denoising, and super-resolution based
on a specially designed deep convolutional neural network (CNN). All previous
works assumed that a noise level is known in advance or properly estimated.

##  3 Framework

###  3.1 StereoISP

We address our problem in the case of a two-camera ISP. Assume that we have
raw image pairs: primary image $M$ and secondary image $S$ such that
$M,S\in\mathbb{R}^{0pt\times 0pt}$ where $0pt$ and $0pt$ are the height and
width of images, respectively. $M$ and $S$ can be summarized by a Poisson
model such that $M$ and $S\sim$ Poisson($\lambda$), where $\lambda$ is the
average total number of photons seen by the sensor [1]. Our framework proposed
that the output denoised image $O$ becomes a function of the primary image and
the warped secondary image with the guidance of the disparity map $D$. Eq. 1
demonstrates the proposed framework:

$O=\mathcal{D_{M}D_{N}}(M,\mathcal{W}(S,D)))$ (1)

where $\mathcal{W}$ is a warping function results projection of secondary
image $S$ onto the primary image $M$ guided by the disparity map $D$.
$\mathcal{D_{M}}$ and $\mathcal{D_{N}}$ are demosaicking and denoising
processes, respectively. So that the primary image and warped image become
concatenated to be passed into demosaicking $\mathcal{D_{M}D_{N}}$. The output
denoised image $O\in\mathbb{R}^{3\times p\times r}$ is an RGB image.

For an end-to-end solution, the disparity map $D$ can be considered as a
stereo matching mapping function $g$ exploits the primary and secondary images
to perform disparity map $D$, i.e., $D=\mathcal{G}(M,S)\in\mathbb{R}^{p\times
r}$. The output $O$ can be re-written as a function of primary and secondary
images:

$O=\mathcal{D_{M}D_{N}}(M,\mathcal{W}(S,\mathcal{G}(M,S))))$ (2)

For generalization purposes, Fig. 1(b) illustrates the end-to-end novel ISP
formulated in Eq. 2. For studying the feasibility of the idea and
investigating its practicality, we will continue our discussion based on Eq. 1
to study the upper bound of the framework.

###  3.2 Stereo DemosaicNet

(a) left image (ground truth)

(b) Left raw image

(c) Estimated disparity map from raw images

(d) Right image (ground truth)

(e) Right raw image

(f) Ground truth disparity map

Figure 3: Disparity estimation using modified PSMNet [2] with KITTI 2015 [20]
by using raw stereo images.

(a) left image (ground truth)

(b) Left raw image

(c) Estimated disparity map from raw images

(d) Right image (ground truth)

(e) Right raw image

(f) Ground truth disparity map

Figure 4: Disparity estimation using modified PSMNet [2] with KITTI 2015 [20]
by using raw stereo images.

Inspired by [7], we extend on DemosaicNet that is originally designed to
obtain denoised RGB images using a single image. In Stereo DemosaicNet, we use
a standard feed-forward architecture to implement our stereo operator that has
been illustrated in Fig. 2. Our network is composed of $N+3$ convolutional
layers. We denote W as the number of outputs per input. In the case of the
dual camera model, each convolutional layer has 2W outputs using the kernel
size $K\times K$, followed by batch normalization layer to prevent the problem
of internal covariate shift [11].

We first preprocess a warped image by applying backwarping on the secondary
image guided by the disparity map. With the guidance of disparity map $D$, we
warp the secondary image to the primary image by creating a rectilinear grid
with size $p\times r$. We set the interpolator to interpolate the locations of
warped images linearly. We denote the output of the warping process as $W$. We
rearrange the samples of the Bayer input mosaic to obtain multi-frame multi-
channel images and concatenate them together. This process makes the spatial
pattern translation invariant with a period of 1 pixel reducing the
computational cost in the following steps.

After we prepared the data to be passed into the denoising operator (in this
case it is stereo DemosaicNet), we pack the primary image and the warped image
together, which results in an 8-channel layer $F^{0}$. We assume the noise
level of input images is known or has been computed by a method that estimates
the noise level. Since a combination of Poisson and Gaussian noises in linear
space accurately models camera noise, we use the Poisson Gaussian noise model
in this pipeline [6]. We added a 9-th channel to the stack of $F^{0}$.

The bulk of the feature extraction process is performed at this lower
resolution by the next $N$ blocks. Each block consists of a filter bank of
spatial footprints $K\times K$ followed by a batch normalization layer. The
output of batchNorm passes through a point-wise ReLU non-linearity $f(\cdot)=$
max(0,$\cdot$). The last layer in low-resolution feature extraction, $F^{N}$,
has 24 channels that correspond to the color samples of a $2\times 2$
neighborhood.

We upsample the output of $F^{N}$ to a full resolution in $F^{N+1}$. It should
be noted that the first 12 channels correspond to the primary image $M$,
whereas the last 12 channels correspond to warped input $W$. We split the
upsamples output and concatenate masked copies of $M$ and $W$ to their
corresponding upsampled features. We stack the split layers where we place the
masked input and its corresponding features extracted in the front of
$F^{N+2}$ which results in a 12 channels layer. For full-resolution
reconstruction, we feed $F^{N+2}$ into a full-resolution convolutional
features layer followed by a batch normalization operator and a point-wise
ReLU non-linearity function. The final output $O$ on the network is an affine
combination of the last feature maps that contain features maps extracted on
the lower resolution and forward masked inputs $F^{N+3}$. We opted for a
stereo DemosaicNet with W = 15 and D = 64 per input, which is similar to the
original work [7].

##  4 Experiments

###  4.1 Training procedure

In this experiment, assume that the disparity map between stereo pairs is
calculated by one of the methods mentioned in Sec. 2, so we use the ground
truth disparity maps. We use a dataset
$\mathcal{D}=\\{(\mathrm{M}_{i},\mathrm{S}_{i},\mathrm{D}_{i},\mathrm{I}_{i})\\}_{i}$
where $\mathrm{M}_{i}$ is the mosaicked and noisy version of an $i$-th example
of RGB image $I_{i}$. We fix $\sigma$ for the training procedure to be 10,
where $\sigma$ is the average total number of photons seen by the sensor as
mentioned in Sec. 3, We optimize the weights and biases by minimizing $L_{2}$
norm on the training set:

$\mathcal{L}\left(\left\\{w^{(n)},b^{(n)}\right\\}_{n}\right)=\frac{1}{(p\times
r)|\mathcal{D}|}\sum_{i}\left\|\mathrm{O}_{i}-\mathrm{I}_{i}\right\|^{2}$ (3)

such that $w^{(n)}$ and $b^{(n)}$ are weights and biases of $n$-th layer,
respectively, and $p\times r$ is the patch size for the training samples. We
assess our measurement by utilizing peak signal-to-noise ratio (PSNR) to
measure the quality of output image $O_{i}$ with respect to the ground truth
$\mathrm{I}_{i}$. Note that the maximum value for RGB is 255 per channel as
each pixel consists of 8 bits.

$\mathrm{PSNR}=20\log_{10}\left(\frac{\mathrm{MSE}_{O_{i}}}{\sqrt{\mathrm{MAX}_{I_{i}}}}\right)$
(4)

We exploited KITTI 2015 [20] that contains 200 pairs, where we split the
dataset into 160 training examples and 40 testing examples. to improve image
quality, we also considered a large-scale dataset for stereo matching
drivingStereo [25] that contains 174,437 training pairs and 7,751 testing
pairs. To the best of our knowledge, there is no raw stereo image dataset in
the literature. Therefore, we create the mosaicked and noisy images $M$ and
$S$ by retaining only one color channel per pixel according to the Bayer
pattern from the sRGB images [7].

The training has been done in two stages: in the first stage, we pass
$\mathrm{M}_{i}$ twice with different noise structures in both input ports
(i.e., $M$ and $W$). The reason is that we want to let the model perform
denoising tasks and demosaicing with the ideal disparity between primary and
secondary images. That means the projection of the secondary image is
identical to the primary image except for the effect of noise. After the model
is converged, we replace the input of the $W$ port with a warping block with
the sake of letting the model inherits the information of the objects in the
secondary image and being added to the primary image which results increasing
in the PSNR.

In all experiments, we use a patch size $p=256$ by $r=512$ pixels and we
initialized all biases $b^{(n)}$ to 0. We fixed the noise level $\sigma=10$
such that the noise follows Poisson distribution which depicts shot noise. The
optimizer is carried out by ADAM [13]. We use a batch size to be 8 and an
initial learning rate of $10^{-4}$. The training is performed with PyTorch on
a single NVIDIA V100 GPU with 16 GB memory.

###  4.2 Stereo with raw images

Since stereo-matching for raw images has not been yet well studied, we aim to
investigate current stereo-matching architecture to assess the feasibility to
do disparity estimation tasks in the raw domain. We use PSMNet [2] on KITTI
2015 dataset. We present our results for disparity map estimation on raw
stereo images and exploiting PSMNet [2]. Although results needed to be
improved, it seems it is promising to develop the feature extraction modules
in stereo matching architecture to handle demosaicing/denoising artifacts s
shown in figures 3 and 4. As our goal in this paper is to prove our hypothesis
that we want to investigate the feasibility to inherit more information from
the secondary image, our discussion in the following section will focus on
using the ground truth disparity map.

###  4.3 DemosaicNet vs Stereo DemosaicNet

| KITTI 2015 | drivingStereo
---|---|---
Method | PSNR (dB)
DemosaicNet [7] | 24.47 | 29.59
Ours (Sec. 3.2) | 26.79 (+9.48%) | 32.02 (+8.2%)
Table 1: Evaluation on KITTI 2015 [7] and drivingStereo [25] our proposed
model architecture based on proposed StereoISP framework.

Figure 5: Evaluation on KITTI 2015 [20] dataset with dense disparity map. From
left to right: left mosaicked noisy image $M$, right mosaicked noisy image
$S$, DemosaicNet [7], ours proposed in Sec. 3.2, ground truth $I$.

Figure 6: Evaluation on drivingStereo [25] dataset with sparse disparity map.
From left to right: left mosaicked noisy image $M$, right mosaicked noisy
image $S$, DemosaicNet [7], ours proposed in Sec. 3.2, ground truth $I$.

To assess our hypothesis, we make use of DemosaicNet [7] as a benchmark for
our model since ours is an extended version of DemosaicNet. We make use of
DemosaicNet architecture PyTorch implementation by the author
111https://github.com/mgharbi/demosaicnet. We extended on original DemosaicNet
implementation by adding a noise layer in $F^{0}$. Since Authors in [7]
trained their architecture on carefully selected dataset, we no longer be able
to replicate their results. However, we consider two automobile datasets:
KITTI 2015 [20] and a large-scale drivingStereo [25]. Because of the nature of
stereo automobile datasets as they include color bias (i.e., almost half of
the image is a sky where its color is either blue or white or both), and given
that we do not have as sufficient data in the stereo case as in other
datasets, especially in automobile datasets, we expect DemosaicNet results on
KITTI 2015 and drivingStereo could be lower than what authors stated in their
work.

We trained the original model on KITTI 2015 for 2,000 epochs, with a decaying
learning rate by a factor of 10 every 100 epochs. The model converged with a
validation PSNR of 24.47 dB. We trained Stereo DemosaicNet in two stages as
explained in Sec. 4.1 with 2,246,146 trainable parameters. Our model converged
with validation PSNR of 26.79 dB, with an improvement in the quality of the
reconstructed image by 9.45%. Results obtained by DemosaicNet and our model
have been presented in Fig. 5. We considered drivingStereo [25] with large-
scale stereo images. As we trained the model for the large-scale dataset. The
model has converged with a validation PSNR of 29.59 dB. We trained stereo
DemosaciNet and we reached a validation PSNR of 32.02 dB, with an improvement
in the quality of the reconstructed image by 8.2%. Results for drivingStereo
dataset have been shown in Fig. 6. Table 1 summarizes results and comparisons
between DemosaicNet and our model inspired by the StereoISP framework.

###  4.4 Ablation Study

####  4.4.1 Same input with different noise

Figure 7: Study the upper bound of StereoISP frame work on KITTI 2015 [20]. We
input same images with different structured noise in a place of $M$ and $W$ in
stereo DemosaicNet, i.e., the ideal case where disparity between two images is
zero. From left to right: primary images $M$, warped image at disparity map =
0, denoised image, ground truth, error in reconstructions.

Figure 8: Study the upper bound of StereoISP frame work on drivingStereo [25].
We input same images with different structured noise in a place of $M$ and $W$
in stereo DemosaicNet, i.e., the ideal case where disparity between two images
is zero. From left to right: primary images $M$, warped image at disparity map
= 0, denoised image, ground truth, error in reconstructions.

We assume that we can get the maximum quality of the image if the disparity
between the two images equals 0. Therefore, we trained Stereo DemosaicNet to
find out the upper bound of the model. We corrupt the same image with
different noise structures with the same noise level $\sigma$. The model
successfully performed demosaicing and denoising as shown in figures 7 and 8.
It shows that an error between two images has appeared only on the edges of
objects. Although this case is not realistic as shot noise happened in
cameras, it became useful to test the maximum image quality that the model can
provide.

####  4.4.2 Image-pair without warping

| KITTI 2015 | drivingStereo
---|---|---
Method | PSNR (dB)
Baseline [7] | 24.47 | 29.59
Unwarped (Sec. 4.4.2) | 24.37 (-0.41%) | -
Ours (Sec. 3.2) | 26.79 (+9.48%) | 32.02 (+8.2%)
Same (Sec. 4.4.1) | 27.38 (+11.89%) | 32.61 (+10.21%)
Table 2: Ablation study Stereo DemosaicNet model based on StereoISP framework.

We study the effect of image warping block $\mathcal{W(\cdot)}$ by removing it
and passing a stereo-pair directly to the stereo demosaicing/denoising module
$\mathcal{D_{M}D_{N}}$. In this case, the module is responsible to perform
three major tasks: (1) estimate the disparity between the image-pair. (2) Warp
the secondary image onto the primary image. (3) integrate the warped image
with the primary image. As stereo DemosaicNet was designed to handle denoising
tasks, it was hard to achieve other tasks. In table 2, we quantified the
performance of the ablation study and our proposed model. It is not surprising
that feeding the stereo-pair into the model not only does not change the image
quality, but it degrades the performance below the baseline model [7].

## References

  * [1] Stanley H Chan.  What does a one-bit quanta image sensor offer?  IEEE Transactions on Computational Imaging, 8:770–783, 2022.
  * [2] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410–5418, 2018.
  * [3] Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom Drummond, and Zongyuan Ge.  Hierarchical neural architecture search for deep stereo matching.  Advances in Neural Information Processing Systems, 33, 2020.
  * [4] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  * [5] Thibaud Ehret, Axel Davy, Pablo Arias, and Gabriele Facciolo.  Joint demosaicking and denoising by fine-tuning of bursts of raw images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8868–8877, 2019.
  * [6] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian.  Practical poissonian-gaussian noise modeling and fitting for single-image raw-data.  IEEE Transactions on Image Processing, 17(10):1737–1754, 2008.
  * [7] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand.  Deep joint demosaicking and denoising.  ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.
  * [8] Clément Godard, Kevin Matzen, and Matt Uyttendaele.  Deep burst denoising.  In Proceedings of the European conference on computer vision (ECCV), pages 538–554, 2018.
  * [9] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1712–1722, 2019.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916, 2015.
  * [11] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [12] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In Proceedings of the IEEE international conference on computer vision, pages 66–75, 2017.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [14] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.  Noise2void-learning denoising from single noisy images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2129–2137, 2019.
  * [15] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  arXiv preprint arXiv:1803.04189, 2018.
  * [16] Dasong Li, Yi Zhang, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li.  Efficient burst raw denoising with variance stabilization and multi-frequency denoising network.  arXiv preprint arXiv:2205.04721, 2022.
  * [17] Zhetong Liang, Shi Guo, Hong Gu, Huaqi Zhang, and Lei Zhang.  A decoupled learning scheme for real-world burst denoising from raw images.  In European Conference on Computer Vision, pages 150–166. Springer, 2020.
  * [18] Ziwei Liu, Lu Yuan, Xiaoou Tang, Matt Uyttendaele, and Jian Sun.  Fast burst images denoising.  ACM Transactions on Graphics (TOG), 33(6):1–9, 2014.
  * [19] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040–4048, 2016.
  * [20] Moritz Menze and Andreas Geiger.  Object scene flow for autonomous vehicles.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3061–3070, 2015.
  * [21] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll.  Burst denoising with kernel prediction networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2502–2510, 2018.
  * [22] Guocheng Qian, Yuanhao Wang, Jinjin Gu, Chao Dong, Wolfgang Heidrich, Bernard Ghanem, and Jimmy S. Ren.  Rethinking learning-based demosaicing, denoising, and super-resolution pipeline.  In 2022 IEEE International Conference on Computational Photography (ICCP), pages 1–12, 2022.
  * [23] Xuejian Rong, Denis Demandolx, Kevin Matzen, Priyam Chatterjee, and Yingli Tian.  Burst denoising via temporally shifted wavelet transforms.  In European Conference on Computer Vision, pages 240–256. Springer, 2020.
  * [24] Wenzhu Xing and Karen Egiazarian.  End-to-end learning for joint image demosaicing, denoising and super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3507–3516, 2021.
  * [25] Guorun Yang, Xiao Song, Chaoqin Huang, Zhidong Deng, Jianping Shi, and Bolei Zhou.  Drivingstereo: A large-scale dataset for stereo matching in autonomous driving scenarios.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [26] Jure Zbontar, Yann LeCun, et al.  Stereo matching by training a convolutional neural network to compare image patches.  J. Mach. Learn. Res., 17(1):2287–2318, 2016.
  * [27] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE transactions on image processing, 26(7):3142–3155, 2017.