# Designing Machine Learning Pipeline Toolkit for AutoML Surrogate Modeling
Optimization

Paulito P. Palmes
IBM Research
Mulhuddart, Dublin 15, Ireland
<EMAIL_ADDRESS>
Akihiro Kishimoto
IBM Research
Mulhuddart, Dublin 15, Ireland
<EMAIL_ADDRESS>Radu Marinescu
IBM Research
Mulhuddart, Dublin 15, Ireland
<EMAIL_ADDRESS>
Parikshit Ram
IBM Research
Sandy Springs, GA, United States
<EMAIL_ADDRESS>
Elizabeth Daly
IBM Research
Mulhuddart, Dublin 15, Ireland
<EMAIL_ADDRESS>
###### Abstract

The pipeline optimization problem in machine learning requires simultaneous
optimization of pipeline structures and parameter adaptation of their
elements. Having an elegant way to express these structures can help lessen
the complexity in the management and analysis of their performances together
with the different choices of optimization strategies. With these issues in
mind, we created the AutoMLPipeline (AMLP) toolkit which facilitates the
creation and evaluation of complex machine learning pipeline structures using
simple expressions. We use AMLP to find optimal pipeline signatures, datamine
them, and use these datamined features to speed-up learning and prediction. We
formulated a two-stage pipeline optimization with surrogate modeling in AMLP
which outperforms other AutoML approaches with a 4-hour time budget in less
than 5 minutes of AMLP computation time.

##  1 Introduction

The typical machine learning (ML) workflow for classification and prediction
requires some or a combination of the following preprocessing steps:

  * •

_data cleaning_ : Imputation, Interpolation, etc.

  * •

_feature transformation_ : Normalization, Scaling, One-hot encoding, etc.

  * •

_feature selection_ : Anova, Correlation, etc.

  * •

_feature extraction_ : PCA, ICA, FactorAnalysis, etc.

  * •

_modeling_ : RandomForest, XGBoost, SVM, AdaBoost, etc.

Each step has several choices of functions to use, together with their
corresponding parameters to initialize. Optimizing the performance of the
entire pipeline is a combinatorial search for the proper order and combination
of preprocessing steps, optimization of their corresponding parameters, and a
search for the optimal model and its hyper-parameters.

Because of close dependencies among these various steps to perform
optimization, the entire process is commonly called combined algorithm
selection and hyper-parameter optimization or CASH [1; 2]. Having an elegant
way to express pipeline structures can help lessen the complexity in the
management and analysis of the wide-array of choices of optimization routines.
AMLP (https://github.com/IBM/AutoMLPipeline.jl) toolkit aims to address this
issue by supporting the following features:

  * •

Pipeline API that allows high-level description of modeling and preprocessing
workflow to support explainability and easy dataminining of optimal pipeline
signatures

  * •

Symbolic pipeline parsing to facilitate easy expression of complex pipeline
structures

  * •

Common API wrappers for ML libs including _scikit-learn_ [3], _caret_ [4],
etc.

  * •

Easily extensible architecture by overloading two interfaces: fit! and
transform!

  * •

High-level implementation of meta-ensembles for the ensemble composition of
ensembles (recursively if needed) for robust prediction routines.

  * •

Categorical and numerical feature selectors for specialized preprocessing
routines based on types.

  * •

High-level code of parallelized cross-validation (CV) routines by leveraging
on the multi-threading and distributed computing features in _Julia_ [5].

AMLP’s main feature is the use of relatively compact symbolic expression in
pipeline composition. For instance, a pipeline expression to extract the
numerical features (numf) for pca decomposition, concatenated with one-hot
encoding (ohe) of categorical features (catf) of a given data for RandomForest
(rf) modeling can be written as:

⬇

pipe = @pipeline (numf |> pca) + (catf |> ohe) |> rf

AMLP is written in _Julia_ language [5] to leverage on the latter’s support of
modern features such as: multiple dispatch, just-in-time (_JIT_) compilation,
multi-threading, parallel and distributed computing, dynamic types,
coroutines, interactive shell, high-performance, code specialization,
metaprogramming, and type inference. Pure _Julia_ ML models in AMLP is easy to
maintain and extend by relying on just one programming language because
Julia’s _JIT_ avoids the need to implement some performance-critical tasks in
C/C++.

###  1.1 Related Work

AMLP’s main design was inspired by the _Orchestra_ [6] package. Early ML
packages for _Julia_ are wrappers of existing ML toolkits from _scikit-learn_
in _Python_ [7] and _caret_ in _R_ [8]. By having common APIs across different
ML implementations, _Orchestra_ showed that one can mix and match ML
preprocessing routines from _caret_ , _scikit-learn_ , and _Julia_ seamlessly
in a pipeline by leveraging on _PyCall_ [9] and _RCall_ [10] wrappers for
_Python_ and _R_ , respectively. This is a convenient package because there
are many cases where implementations of ML functions can be found exclusively
in either _scikit-learn_ or _caret_. Having both libraries available for
perusal is a great productivity boost for machine learners in _Julia_.
Unfortunately, _Orchestra_ package has not been maintained for more than 6
years, but its legacy lives in the AMLP’s design. Another popular toolkit in
_Julia_ ML ecosystem is the _MLJ_ [11] package developed at the Alan Turing
Institute. It serves as a meta-package of ML libraries both native to _Julia_
as well as _scikit-learn_.

IBM 111IBM and the IBM logo are trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of
IBM trademarks is available on http://ibm.com/trademark. _Lale_ [12] is a
toolkit in _Python_ with sophisticated support of pipeline optimization useful
for AutoML [13] algorithm development. _Lale_ builds on _scikit-learn_ and
extends it by supporting features such as: automation, correctness checks, and
interoperability. _Lale_ has consistent high-level interface for popular
AutoML algorithms such as: _Hyperopt_ [14; 15], _GridSearchCV_ [3], and _SMAC_
[16]. _Lale_ makes it possible to have an easy comparison of the different
AutoML algorithms for benchmarking, and research, as well as for applications.
While AMLP is at the early stage of development, its pipeline architecture can
be used as building blocks for the development of future AutoML algorithms in
_Julia_ ’s ecosystem. For instance, we were able to create a _Julia_ wrapper
of _Lale_ (https://github.com/IBM/Lale.jl) in less than a week of work by
adapting and extending AMLP’s type hierarchy and interfaces using multiple
dispatch and operator overloading to emulate the pipeline grammar in _Lale_.

##  2 AMLP Architecture

The code in Listing 1 describes the abstract type hierarchy used in the AMLP
toolkit. At the top of the hierarchy is the Machine abstraction with abstract
functions fit! and transform! to be implemented by its subtypes. Calling both
functions in sequence are handled by fit_transform!. A Machine has two
subtypes: Computer and Workflow. A Computer can either be a Transformer or a
Learner while a Workflow can be either a Pipeline or ComboPipeline.

⬇

1abstract type Computer <: Machine

2abstract type Workflow <: Machine

3abstract type Learner <: Computer

4abstract type Transformer <: Computer

5

6function fit!(mc::Machine, input::DataFrame, output::Vector)

7 error(typeof(mc)," has no implementation.")

8end

9

10function transform!(mc::Machine, input::DataFrame)

11 error(typeof(mc)," has no implementation.")

12end

13

14function fit_transform!(mc::Machine, input::DataFrame, output::Vector)

15 fit!(mc,input,output)

16 transform!(mc,input)

17end

Listing 1:  Abstract Type Hierarchy

All Machine subtypes are expected to define their own fit! and transform!
before they can be part of the elements in a Workflow. Any Workflow instance
uses these two functions during training, feature transformation, prediction,
and cross-validation (CV). By convention in _Julia_ [5], functions ending with
exclamation mark (!) mutate the value(s) of their arguments which is done by
fit! and transform! to mutate the states and parameters of the Machine
instance in their arguments.

The AMLP workflow is based on the _Orchestra_ [6] package which drew its
inspiration from the _Unix_ pipeline [17]. The main elements of a pipeline are
a series of Computer instances with each instance performing a specific task.
A typical pipeline for classification or prediction contains a series of
transformers terminated by a learner. During fit_transform!, these series of
transformers act as filters converting the input data into the same
mathematical or statistical transform before feeding them to the learner for
training and prediction. In a pipeline expression where the last element is
not a learner, fit_transform! acts as a feature filter or transformer only.

The Pipeline instance processes linearly the sequence of information among its
elements. Its fit! implementation iteratively calls its elements’
fit_transform! passing the output from one Computer instance to the next
Computer instance in the sequence. Aside from sequential operations, two or
more workflows can be combined using the ComboPipeline instance.

The fit! and transform! functions for a Learner are equivalent to training and
prediction, respectively. A Learner instance implements the algorithm to
acquire the mapping between its input and output during fit! and applies the
learned model to perform prediction during transform!.

For a Transformer, fit! and transform! are preprocessing operations to convert
the training data into the same mathematical or statistical transform.
Depending on the function used, fit! can be a noop (no operation) like in sqrt
or log transform. On the other hand, PCA or ICA uses fit! to compute and store
the coefficient matrix derived from its training input and applies the same
matrix in transform!.

###  2.1 AMLP workflow

The code in Listing 2 depicts the typical usage of the AMLP toolkit. Lines 1--
2 loads the AMLP package and the pro football dataset [18; 19], respectively.
The aim is to predict whether the game is held at _home_ or _away_ based on
the following (C)ategorical or (N)umerical features: FavoritePoints (N),
UnderdogPoints (N), Pointspread (N), FavoriteName (C), UnderdogName (C), Year
(N), Week (N), Weekday (C), and Overtime (C).

⬇

1using AMLP # load package

2(X,Y) = getprofb() # load dataset

3

4# Instantiate the pipeline elements

5pca = SKPreprocessor("PCA")

6mx = SKPreprocessor("MinMaxScaler")

7ohe = OneHotEncoder()

8catf = CatFeatureSelector() # categorical columns

9numf = NumFeatureSelector() # numerical columns

10rf = SKLearner("RandomForestClassifier")

11

12# Setup the ML pipeline

13pipe = @pipeline ((catf |> ohe) + (numf |> mx |> pca)) |> rf

14

15# train and predict

16prediction = fit_transform!(pipe,X,Y)

17

18# compute avg accuracy by 10-fold cv

19performance = crossvalidate(pipe,X,Y)

Listing 2:  AMLP toolkit sample usage

Lines 5--10 of Listing 2 create instances of the preprocessing elements to be
included in the pipeline. The toolkit uses the _scikit-learn_ [3] wrappers,
SKPreprocessor and SKLearner, to instantiate its transformers and learner:
PCA, MinMaxScaler, and RandomForestClassifier. Other elements such as:
OneHotEncoder, CatFeatureSelector, and NumFeatureSelector are implemented in
pure _Julia_.

In line 13 of Listing 2, the expression:

⬇

pipe = @pipeline ((catf |> ohe) + (numf |> mx |> pca)) |> rf

describes the preprocessing workflow of the input data with the RandomForest
classifier as the learner. The expression, (x |> f), is equivalent to f(x)
while the expression, (x + y), signifies feature union: $\cup$(x,y). The
expression, (catf |> ohe), selects columns with categorical features and
transform them into one-hot representation. In (numf |> mx |> pca)), numerical
columns are selected then scaled by MinMaxScaler and finally embedded in PCA
subspace. Both features are then concatenated to become the input features of
the RandomForest model (rf).

Line 13 of Listing 2 can also be written using function calls as:

⬇

Pipeline(ComboPipeline(Pipeline(catf, ohe),

Pipeline(Pipeline(numf, mx), pca)), rf)

This latter expression looks less understandable compared to the former. The
simplicity of AMLP pipeline expression becomes more significant in developing
AutoML algorithms or in data mining optimal pipelines among different datasets
for surrogate modeling [20]. More examples of AMLP usage including its
extensive documentation and source code can be found in its open-source github
resource [to be referenced if accepted].

##  3 Pipeline search strategy benchmark

The search for an optimal pipeline can be treated as a matching problem
between a group of learners and a group of preprocessing pipelines. The
objective is to find the optimal pair of preprocessing pipeline and learner
such that their corresponding CV accuracy is the best among the rest of the
pairs. We refer to any of these pairs as an ML pipeline (MLPL) in contrast to
a preprocessing pipeline (PRPL) where the last element is not a learner. The
time complexity to search for all possible combinations of MLPL is dependent
on the number of learners as well as the size of elements in the PRPL:
n(learners) * n(PRPL).

###  3.1 Two-stage strategies

To avoid the brute-force approach of exhaustive search, we attack the problem
by decomposing the search into two stages:

  * •

_One-all_ search strategy uses an arbitrarily chosen learner as the engine of
CV in searching for the best preprocessing pipeline performance in the first
stage. The second stage proceeds by using the best pipeline found in the first
stage to search for the best learner.

  * •

_All-one_ search strategy uses an arbitrarily chosen pipeline as the base
pipeline to search for the best learner during the first stage. The second
stage uses the best learner found in the first stage to search for the best
pipeline.

We call the first strategy _one-all_ to indicate the utilization of a
surrogate learner to evaluate all pipelines in the first stage. Similarly, we
call the second strategy _all-one_ to indicate the evaluation of all learners
under a surrogate pipeline. To aid the comparison, we also implemented the
_all-all_ strategy which is an exhaustive search of the performance of all
combinations of learners and PRPL to get the best MLPL.

In our implementation, we call the pipeline consisting of scaler (sc) and
feature extractor (fx) a one-block PRPL, expressed as: (sc |> fx). In the
experiments, we use the maximum of two-block PRPL, ((sc1 |> fx1) + (sc2 |>
fx2)), where sc1 != sc2 or fx1 != fx2. The pipeline can easily be extended by
adding more PRPL blocks depending on the complexity of the dataset at the
expense of longer computation time.

###  3.2 Experimental setup

Table 1: OpenML Datasets Dataset | NAs | Size | Cat | Num | Rows | Cols | Class | Best Result | AutoML
---|---|---|---|---|---|---|---|---|---
analcatdata | 0 | 22K | 5 | 0 | 9873 | 5 | 6 |  75.10 $\pm$ 02.59 | ATM
breast-w | 16 | 18K | 1 | 9 | 699 | 10 | 2 |  01.44 $\pm$ 02.22 | ATM
cmc | 0 | 30K | 8 | 2 | 1473 | 10 | 3 |  43.33 $\pm$ 01.52 | TPOT
credit-app | 67 | 39K | 10 | 6 | 690 | 16 | 2 |  11.11 $\pm$ 03.52 | ATM
eucalyptus | 448 | 74K | 6 | 14 | 736 | 20 | 5 |  35.41 $\pm$ 05.02 | AutoSk
first-order | 0 | 2.6M | 1 | 51 | 6118 | 52 | 6 |  38.48 $\pm$ 01.95 | TPOT
GestureSeg | 0 | 3.3M | 1 | 32 | 9873 | 33 | 5 |  32.78 $\pm$ 03.59 | TPOT
jm1 | 25 | 826K | 1 | 21 | 10885 | 22 | 2 |  18.13 $\pm$ 01.55 | TPOT
profb | 1200 | 25K | 5 | 5 | 672 | 10 | 2 |  32.43 $\pm$ 05.84 | TPOT
plants-shape | 0 | 892K | 1 | 64 | 1600 | 65 | 100 |  36.54 $\pm$ 03.53 | AutoSk
sick | 6064 | 300K | 23 | 7 | 3772 | 30 | 2 |  01.30 $\pm$ 00.87 | TPOT
soybean | 2337 | 171K | 36 | 0 | 683 | 36 | 19 |  06.59 $\pm$ 02.79 | ATM

Table 1 summarizes the statistical features of the 12 OpenML datasets [19]
used in the experiment. Also included are the best results in a 4-hour time
budget discussed in the review paper of [21]. The comparison involves 5 AutoML
approaches, namely: TPOT [22], Auto-Sklearn [23; 24], ATM [25], Hyperopt-
Sklearn [14; 15], and Random Search [26]. Among these approaches, TPOT and ATM
have dominated the rankings in terms of best performance based on the CV
classification errors on the 12 datasets as shown in the last two columns.

The AMLP experiments use the following transformers and learners including
Noop (no operation):

  * •

6 scalers: Standard, MinMax, Robust, Normalizer, PowerTransfomer, Noop

  * •

4 feature extractors: PCA, ICA, FactorAnalysis, Noop

  * •

6 learners: RandomForest, AdaBoost, DecisionTree, GradientBoosting, LinearSvm,
RbfSvm

For the one-block PRPL experiment, the size of all MLPL for the exhaustive
search will be $6\times 4\times 6=144$. A relatively small size that can
easily be evaluated to search for the optimal solution which is ideal for
experimental comparison with the two-stage strategies. However, the size of
all combinations of MLPL suddenly explodes to more than 3,000 pipelines by
just adding another PRPL block. With 10-fold CV per MLPL, the exhaustive
search requires 1440 CV operations for a one-block MLPL and more than 30,000
CV operations for a two-block MLPL.

The _one-all_ search strategy requires $6\times 4=24$ one-block PRPL matched
to one learner for the first stage. The second stage requires matching the
best pipeline against 6 learners. In this strategy, a total of 30 MLPL
pipelines are needed to get the optimal or suboptimal solution. With 10-fold
CV per MLPL, _one-all_ requires 300 CV operations which is 4.8x smaller
compared to the 1440 CV operations needed for the exhaustive search strategy.

For the _all-one_ strategy, 6 learners are matched to a single pipeline in the
first stage. The best learner found is then matched to $6\times 4=24$ PRPL in
the second stage. In total, the _all-one_ uses 30 MLPL which translates to 300
CV operations similar to the _one-all_. The actual runtime performance for
_one-all_ and _all-one_ in the same dataset depends mostly on the best learner
performance in _all-one_ and the surrogate learner performance in _one-all_.

All datasets undergo common cleaning workflow in all experiments. The entire
process is summarized in the following:

⬇

basepipeline = @pipeline colnarm |> rownarm |> ((catf |> ohe) + numf)

For each dataset, column-wise removal (colnarm) of missing values (_NA_) is
carried out on those columns with _NA_ count greater than 10% of the row size
followed by row-wise removal (rownarm) of any remaining _NA_ s. After _NA_
removal, categorical features (catf) of the dataset are transformed to one-hot
(ohe) representation and concatenated with its numerical features (numf).

###  3.3 Results and Discussion

Table 2: _All-all_ (one-block) search strategy |  |  |  |  | Block |
---|---|---|---|---|---|---
Dataset | Rank | AvgErr | Std | Time | Sc | Fx | Learner
analcatdata-dmf | 2 | 76.52 | 4.92 | 0.13 | normalizer | noop | rbfsvc
breast-w | 2 | 02.34 | 1.71 | 0.08 | minmax | noop | rbfsvc
cmc | 1 | 42.57 | 3.37 | 0.18 | noop | noop | gb
credit-approval | 2 | 11.42 | 2.60 | 0.09 | normalizer | noop | rf
eucalyptus | 3 | 36.02 | 6.61 | 0.17 | noop | noop | gb
first-order-theo | 1 | 36.61 | 1.44 | 1.27 | noop | noop | rf
GestureSeg | 1 | 31.96 | 1.28 | 1.87 | minmax | noop | rf
jm1 | 1 | 17.97 | 0.61 | 1.84 | stdsc | noop | rf
profb | 1 | 25.59 | 5.15 | 1.84 | stdsc | noop | lsvc
plants-shape | 1 | 30.25 | 3.03 | 4.29 | powertf | pca | rf
sick | 4 | 06.40 | 0.70 | 0.41 | noop | noop | gb
soybean | 5 | 14.62 | 2.48 | 0.27 | robustsc | noop | lsvc
Median | 1.5 | 27.92 | 2.54 | 0.23 |  |  |

Table 2 summarizes the performance of _all-all_ strategy. The metric of
performance is based on the average classification error (_AvgErr_) using
10-fold CV. The ranking is based on its _AvgErr_ performance in comparison to
the 5 AutoML approaches [21] summarized in Table 1. The _all-all_ median
overall rank is 1.5 using one-block pipeline. Its median runtime to perform
one-block exhaustive search is 0.23 hour or 13.8 minutes per dataset. This
result is encouraging because the median runtime is significantly less
compared to the 5 AutoML algorithms with a 4-hour budget. Closely examining
the runtime of each dataset, _plants-shape_ dataset requires 4.29 hours to
finish while the rest require at most 1.87 hours to run. Take note, however,
that the comparison for runtime is not a fair comparison, as we did not have
the time to repeat the experiments done in [21] using the same machine with
our two-stage strategies. Our experiments were conducted using 2017 Model of
MacBook Pro with 2.8 GHz Quad-Core Intel Core i7 and 16 GB of RAM.

The solution can be improved further by searching all two-block pipelines, but
the size of the search space becomes a limiting factor. Ideally, if either
_one-all_ or _all-one_ has similar optimal results with that of _all-all_ ,
the runtime required will be significantly less to attack the two-block MLPL.
The succeeding discussions of the results of experiments aim to find out if
there is a significant runtime saving in either one or both of the two-stage
strategies based on their performance relative to the exhaustive _all-all_
strategy.

Table 3: _One-all_ (one-block) |  |  |  |  | Block |
---|---|---|---|---|---|---
Dataset | Rank | AvgErr | Std | Time | Sc | Fx | Lr
analcatdata-dmf | 3 | 78.42 | 5.77 | 0.35 | minmax | factA | ada
breast-w | 3 | 2.78 | 1.88 | 0.02 | stdsc | pca | rbfsvc
cmc | 4 | 45.76 | 2.94 | 0.02 | norm | noop | gb
credit-approval | 2 | 11.34 | 4.98 | 0.09 | norm | noop | rf
eucalyptus | 3 | 36.79 | 7.61 | 0.09 | noop | noop | rf
first-order-theo | 1 | 36.89 | 1.68 | 0.32 | powertf | noop | rf
GestureSeg | 1 | 32.04 | 1.16 | 0.15 | noop | noop | rf
jm1 | 1 | 18.01 | 0.51 | 0.34 | powertf | noop | rf
profb | 1 | 28.87 | 4.86 | 0.09 | noop | ica | rf
plants-shape | 1 | 30.80 | 4.44 | 0.13 | powertf | pca | rf
sick | 4 | 6.60 | 0.66 | 0.06 | robustsc | pca | rf
soybean | 5 | 15.29 | 4.78 | 0.34 | robustsc | noop | lsvc
Median | 2.50 | 29.84 | 3.69 | 0.11 |  |  |
Table 4: _All-one_ (one-block) |  |  |  |  | Block |
---|---|---|---|---|---|---
Dataset | Rank | AvgErr | Std | Time | Sc | Fx | Lr
analcatdata-dmf | 3 | 77.92 | 3.42 | 0.01 | noop | noop | rbfsvc
breast-w | 2 | 2.34 | 2.08 | 0.01 | robustsc | noop | rbfsvc
cmc | 1 | 42.36 | 4.14 | 0.08 | stdsc | noop | gb
credit-approval | 2 | 11.26 | 3.72 | 0.03 | norm | noop | rf
eucalyptus | 3 | 36.44 | 4.60 | 0.04 | robustsc | noop | rf
first-order-theo | 1 | 36.48 | 2.08 | 0.17 | powertf | noop | rf
GestureSeg | 1 | 32.39 | 1.09 | 0.30 | robustsc | noop | rf
jm1 | 1 | 18.08 | 0.96 | 0.45 | stdsc | noop | rf
profb | 1 | 25.14 | 6.42 | 0.02 | robustsc | noop | lsvc
plants-shape | 1 | 30.35 | 3.86 | 0.12 | stdsc | pca | rf
sick | 4 | 6.54 | 0.49 | 0.07 | robustsc | pca | gb
soybean | 5 | 14.66 | 3.96 | 0.01 | stdsc | noop | rbfsvc
Median | 1.5 | 27.75 | 3.57 | 0.05 |  |  |
Table 5: _All-one_ (two-block) |  |  |  |  | Block 1 | Block 2 |
---|---|---|---|---|---|---|---
Dataset | Rank | AvgErr | Std | Time | Sc1 | Fx1 | Sc2 | Fx2 | Lr
analcatdata | 2 | 75.54 | 4.91 | 0.56 | minmax | noop | powertf | factA | ada
breast-w | 2 | 2.11 | 1.65 | 0.41 | norm | noop | minmax | noop | rbfsvc
cmc | 1 | 42.57 | 3.45 | 2.28 | stdsc | noop | robustsc | noop | gb
credit-app | 1 | 11.03 | 4.59 | 0.50 | stdsc | noop | powertf | factA | lsvc
eucalyptus | 1 | 33.40 | 6.13 | 2.78 | stdsc | noop | noop | factA | gb
first-order | 1 | 36.37 | 2.49 | 5.51 | noop | ica | noop | noop | rf
GestureSeg | 1 | 30.46 | 1.03 | 9.69 | norm | ica | stdsc | noop | rf
jm1 | 1 | 17.67 | 0.79 | 16.62 | robustsc | noop | norm | pca | rf
profb | 1 | 24.69 | 5.29 | 0.61 | stdsc | ica | stdsc | noop | lsvc
plants-shape | 1 | 26.05 | 2.77 | 3.97 | stdsc | factA | norm | pca | rf
sick | 4 | 6.26 | 1.01 | 2.65 | robustsc | pca | noop | noop | gb
soybean | 5 | 13.24 | 2.62 | 0.57 | robustsc | noop | noop | noop | lsvc
Median/Mode | 1.0 | 25.37 | 2.70 | 2.62 | stdsc | noop | noop | noop | rf
Table 6: _All-one with Surrogates_ | Baseline: All-one | Surrogate: Baseline+PRP | Surrogates: Baseline+PRP+LR
---|---|---|---
Dataset | Rank | AvgErr | Std | Time | Rank | AvgErr | Std | Time | Rank | AvgErr | Std | Time
analcatdata | 2 | 75.54 | 4.91 | 0.56 | 2 | 76.67 | 4.24 | 0.07 | 2 | 76.17 | 0.92 | 0.00
breast-w | 2 | 02.11 | 1.65 | 0.41 | 2 | 02.49 | 1.70 | 0.02 | 2 | 02.33 | 2.49 | 0.00
cmc | 1 | 42.57 | 3.45 | 2.28 | 1 | 42.84 | 4.64 | 0.07 | 2 | 44.13 | 3.68 | 0.01
credit-app | 1 | 11.03 | 4.59 | 0.50 | 2 | 11.60 | 4.83 | 0.03 | 2 | 12.08 | 3.80 | 0.04
eucalyptus | 1 | 33.40 | 6.13 | 2.78 | 3 | 36.37 | 7.03 | 0.16 | 2 | 35.63 | 6.27 | 0.22
first-order | 1 | 36.37 | 2.49 | 5.51 | 1 | 36.73 | 2.18 | 0.13 | 1 | 36.63 | 2.07 | 0.28
GestureSeg | 1 | 30.46 | 1.03 | 9.69 | 1 | 31.58 | 1.45 | 0.44 | 5 | 45.19 | 0.96 | 0.76
jm1 | 1 | 17.67 | 0.79 | 16.62 | 1 | 18.01 | 0.86 | 0.23 | 1 | 17.98 | 1.47 | 0.40
profb | 1 | 24.69 | 5.29 | 0.61 | 1 | 24.86 | 6.02 | 0.02 | 1 | 24.41 | 4.42 | 0.04
plants-shape | 1 | 26.05 | 2.77 | 3.97 | 1 | 32.06 | 2.93 | 0.17 | 3 | 44.31 | 5.53 | 0.61
soybean | 5 | 13.24 | 2.62 | 0.57 | 5 | 14.29 | 3.55 | 0.02 | 5 | 14.29 | 4.83 | 0.03
Median | 1.0 | 26.05 | 2.77 | 2.28 | 1.0 | 31.58 | 3.55 | 0.07 | 2.0 | 35.63 | 3.68 | 0.04

Tables 3 and 4 show the performance of _one-all_ and _all-one_ strategies,
respectively, for one-block MLPL. Their ranking is based on their performances
against the 5 AutoML algorithms. Table 3 summarizes the performance of _one-
all_ using the RandomForest surrogate. All other learners tested as surrogate
have similar 2.0 to 2.5 median rank. The _all-one_ strategy in Table 4 uses
the pipeline expression,(((catf |> ohe) + numf)) |> robustsc, as the surrogate
pipeline to search for the best learner in the first stage. The expression
indicates one-hot encoding (ohe) of the categorical features (catf) combined
with the numerical features (numf) and transformed by robust scaling
(robustsc).

Comparing Tables 2, 3 and 4, the _all-one_ strategy median rank of 1.5 is
similar to _all-all_ and significantly better than the 2.5 median of _one-
all_. The _all-one_ median validation error of 27.75 $\pm$ 3.69 is also
slightly better than _all-all_ (27.92$\pm$ 2.54). While both have similar
validation errors, the biggest advantage of _all-one_ is its speed which is
significantly faster than _all-all_. Its median time duration for all 12
datasets is just 0.05 hour (3 minutes) compared to the runtime median of 0.23
hour (13.8 minutes) for _all-all_. Furthermore, the dataset with worst runtime
is the _analcadata-dmf_ which requires 0.35 hour or 21 minutes. This worst
runtime is still significantly less than the 4-hour budget allotted to the 5
AutoML algorithms. It is interesting to note that most solutions in _all-one_
do not employ any feature extraction but only scaling. We can consider the
solutions in _all-one_ to be sparsed relative to the dominant presence of noop
in the feature extraction part of the one-block MLPL.

Inspired by these promising results, the next experiment applies the _all-one_
strategy to two-block MLPL with the results summarized in Table 5. The _all-
one_ strategy achieves a 1.0 median rank which suggests that its performance
is at least as good or better than the best results of the 5 AutoML algorithms
with respect to the 12 datasets. While the runtime median of _all-one_ is 2.62
hours, 3 datasets require more than 4 hours to finish. In the future, it will
be interesting to incorporate the time budget into the _all-one_ strategy and
run the other AutoMLs in the same machine to have fair comparison.

Using the _all-one_ strategy, there is a trade-off in its applications to one-
block or two-block pipelines. The former has significantly quick runtime but
may not be optimal, while the latter may require relatively longer runtime
than the former for some datasets but with a higher likelihood of being
optimal.

Another interesting observation in Tables 2, 4, and 5 regarding the
composition of their best solutions is the high occurrence of noop. We can
consider their solutions to be sparsed because in many cases they do not fully
utilize the 2 out 5 non-noop scalers and 2 out of 3 non-noop feature
extractors in their optimal pipelines. This insight can be valuable by using
the datamined signatures of the optimal solutions to predict structure
complexity of the pipeline for an efficient search. By mapping data
metafeatures and the corresponding optimal pipeline signatures, we can train a
metalearner to guide which subset of pipelines or elements of the pipelines
can be used as a starting point in search.

Table 6 summarizes the results of implementing these insights using PRP and LR
surrogate models. The PRP-surrogate is trained to learn the mapping between
dataset metafeatures and its corresponding optimal pipeline signature
complexity. Pipeline complexity has 4 categories based on the presence of
_noop_. Category 1 has zero or one _noop_ , category 2 has two _noops_ , etc.
More _noops_ imply less pipeline complexity. On the other hand, the LR-
surrogate is used to learn the mapping between the dataset metafeatures with
its corresponding optimal learner type: Ensemble vs SVM. Both surrogate models
are trained using the OpenML-CC18 [27] datasets by extracting their
metafeatures using _pymfe_ [28]. The extracted datasets, results, and julia-
based pseudocode for surrogate modeling can be found in the supplementary
material submission of the paper.

We use the _all-one_ strategy as the baseline and extended it with PRP and LR
surrogates to find out their effect in prediction error and computation time.
While there is a 5% increase in error by using PRP-surrogate, the median
computation time went down from 2.28 hours to just 4.2 minutes. In spite of
the 5% drop in accuracy, PRP-surrogate median rank of 1.0 still indicates
superior performance relative to other AutoML approaches.

The exponential reduction in computation time of PRP-surrogate is due to the
use of smaller search space due to the removal of unnecessary preprocessing
elements. Incorporating further the LR-surrogate increases the median
prediction error by another 4% but reduces the median computation time to just
2.4 minutes. The main trade-off in relying on more surrogates is the reduction
of computation time at the expense of less accurate prediction. Depending on
the requirements, using one or more surrogates can be necessary to speed-up
computation if the corresponding drop in prediction accuracy is acceptable
which can be true in some application domains.

Among the datasets, the two-stage strategies have consistent low performance
in both _sick_ and _soybean_ problems. Both datasets are characterized by
large number of missing values compared to the rest of the datasets. There are
6064 _NA_ s in _sick_ and 2337 _NA_ s in _soybean_. The poor performance of
the two-stage strategies can be attributed to the non-implementation of
imputation or interpolation filter in the basepipeline data cleaning routine.
Future experiments will examine which imputation or interpolation routines can
be used to improve AMLP solutions.

##  4 Conclusion

Based on the set of experiments we conducted, the _all-one_ strategy provides
the optimal solution in a significantly shorter duration relative to other
AutoML algorithms. The results indicate that using a simple pipeline in the
first stage consisting of one-hot encoded categorical features together with
their numerical features under robust scaling provides a good representation
of the dataset difficulty for evaluating the best matched optimal learner from
the group of learners. The winning learner picked by the first stage can then
be used to improve the solution further by looking for a more optimal match
with the rest of the pipelines. The _all-one_ strategy computation speed can
be exponentially reduced by utilizing PRP and LR surrogates in exchange of
lower accuracy which remains competitive relative to other AutoML approaches.

The easy and straightforward experimental setup in the implementation of these
series of experiments can be attributed to the usability of _Julia_ and AMLP
toolbox. AMLP’s support for a high-level description of the pipelines makes it
easy to track which of the pipeline elements are dominant among the optimal
solutions. These insights can be used in the development of the runtime search
strategy in future algorithms. Data mining optimal pipelines become much
easier because one can directly use these high-level expressions to perform
text-mining, frequency pattern mining, text associations, and NLP on the
elements and structure of the optimal pipeline solutions.

The high-level and easy composability in AMLP can result in the creation of
complex pipelines that can barely be understood. For similar reasons that
regularization and information criterion are employed in mathematical and
statistical modeling, any AutoML implementation relying on AMLP and similar
toolkits must incorporate strategies that take into account a good balance
between pipeline complexity, accuracy, and explainability for easy scrutiny,
comprehensibility, and verification bias of its solutions.

## References

  * [1] M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter.  Efficient and robust automated machine learning.  In Advances in Neural Information Processing Systems, pages 2962--2970, 2015.
  * [2] C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms.  In KDD, pages 847--855, 2013.
  * [3] F. Pedregosa, G. Varoquaux, and A. Gramfort.  Scikit-learn: Machine learning in python.  Journal of Machine Learning Research, 12(1):2825--2830, 2011.
  * [4] Max Kuhn.  Building predictive models in r using the caret package.  Journal of Statistical Software, Articles, 28(5):1--26, 2008.
  * [5] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah.  Julia: A fresh approach to numerical computing.  SIAM review, 59(1):65--98, 2017.
  * [6] S. Jenkins.  Orchestra: Heterogeneous ensemble learning for julia.  https://github.com/svs14/Orchestra.jl, 2014.
  * [7] Guido Van Rossum and Fred L Drake Jr.  Python reference manual.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * [8] R Core Team.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing, Vienna, Austria, 2013.
  * [9] Steven G. Johnson.  PyCall: Calling python functions from the julia language.  https://github.com/JuliaPy/PyCall.jl, 2013.
  * [10] Douglas Bates, Randy Lai, and Simon Byrne.  RCall: Call r from julia.  https://github.com/JuliaInterop/RCall.jl, 2015.
  * [11] Anthony Blaom, Franz Kiraly, Thibaut Lienart, and Sebastian Vollmer.  alan-turing-institute/mlj.jl: v0.5.3, November 2019.
  * [12] Martin Hirzel, Kiran Kate, Avraham Shinnar, Subhrajit Roy, and Parikshit Ram.  Type-driven automated learning with Lale.  CoRR, abs/1906.03957, May 2019.
  * [13] S. Liu, P. Ram, D. Vijaykeerthy, D. Bouneffouf, G. Bramble, H. Samulowitz, D. Wang, A. Conn, and A. Gray.  An ADMM based framework for AutoML pipeline configuration.  In AAAI, 2020.  Their preprint available at https://arxiv.org/pdf/1905.00424.pdf.
  * [14] J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl.  Algorithms for hyper-parameter optimization.  In Advances in Neural Information Processing Systems (NIPS), pages 2546--2554, 2011.
  * [15] J. Bergstra and Y. Bengio.  Random search for hyper-parameter optimization.  Journal of Machine Learning Research, 13(1):281--305, 2012.
  * [16] F. Hutter, H. Hoos, and K. Leyton-Brown.  Sequential model-based optimization for general algorithm configuration.  In International Conference on Learning and Intelligent Optimization (LION), pages 507--523, 2011.
  * [17] Brian W. Kernighan and Rob Pike.  The UNIX Programming Environment.  Prentice-Hall, 1984.
  * [18] H. Stern and R Lock.  Pro football scores.  http://lib.stat.cmu.edu/datasets/profb, 2014.
  * [19] J. Vanschoren, J.N. van Rijn, B. Bischl, and L. Torgo.  Openml: Networked science in machine learning.  SIGKDD Expl., 15(2):49--60, 2013.
  * [20] Tien-Dung Nguyen, Tomasz Maszczyk, Katarzyna Musial, Marc-Andre Zöller, and Bogdan Gabrys.  Avatar - machine learning pipeline evaluation using surrogate model.  In Advances in Intelligent Data Analysis XVIII, pages 352--365. Springer International Publishing, 2020.
  * [21] M.A. Zoller and M. F. Huber.  Survey on automated machine learning.  https://arxiv.org/pdf/1904.12054.pdf, 4 2019.
  * [22] R. Olson, N. Bartley, R. Urbanowicz, and J. Moore.  Evaluation of a tree-based pipeline optimization tool for automating data science.  In Genetic and Evolutionary Computation Conference (GECO), pages 485--492, 2016.
  * [23] M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter.  Auto-sklearn: Efficient and robust automated machine learning, volume 8, chapter Challenges in Machine Learning -- Methods, Systems, Challenges, pages 113--134.  Springer, 2019.
  * [24] M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter.  Auto-sklearn: Efficient and robust automated machine learning.  In NeurIPS, pages 2962--2970, 2015.
  * [25] Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, and Kalyan Veeramachaneni.  Atm: A distributed, collaborative, scalable system for automated machine learning.  IEEE International Conference on Big Data, pages 151--162, 2017\.
  * [26] Richard Loree Anderson.  Recent advances in finding best operating conditions.  Journal of the American Statistical Association, 48(264):789--798, 1953.
  * [27] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren.  Openml benchmarking suites, 2019.
  * [28] Edesio Alcobaça, Felipe Siqueira, Adriano Rivolli, Luís P. F. Garcia, Jefferson T. Oliva, and André C. P. L. F. de Carvalho.  Mfe: Towards reproducible meta-feature extraction.  Journal of Machine Learning Research, 21(111):1--5, 2020.


# DDR-Net: Learning Multi-Stage Multi-View Stereo With Dynamic Depth Range

Puyuan Yi equal contribution Shengkun Tang11footnotemark: 1 Jian Yao
School of Remote Sensing and Information Engineering, Wuhan University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

To obtain high-resolution depth maps, some previous learning-based multi-view
stereo methods build a cost volume pyramid in a coarse-to-fine manner. These
approaches leverage fixed depth range hypotheses to construct cascaded plane
sweep volumes. However, it is inappropriate to set identical range hypotheses
for each pixel since the uncertainties of previous per-pixel depth predictions
are spatially varying. Distinct from these approaches, we propose a Dynamic
Depth Range Network (DDR-Net) to determine the depth range hypotheses
dynamically by applying a range estimation module (REM) to learn the
uncertainties of range hypotheses in the former stages. Specifically, in our
DDR-Net, we first build an initial depth map at the coarsest resolution of an
image across the entire depth range. Then the range estimation module (REM)
leverages the probability distribution information of the initial depth to
estimate the depth range hypotheses dynamically for the following stages.
Moreover, we develop a novel loss strategy, which utilizes learned dynamic
depth ranges to generate refined depth maps, to keep the ground truth value of
each pixel covered in the range hypotheses of the next stage. Extensive
experimental results show that our method achieves superior performance over
other state-of-the-art methods on the DTU benchmark and obtains comparable
results on the Tanks and Temples benchmark. The code is available at
https://github.com/Tangshengku/DDR-Net.

(a) CasMVSNet [10]

(b) UCSNet [5]

(c) DDR-Net (Ours)

Figure 1: Point clouds reconstructed by state-of-the-art methods [5, 10] and
our proposed DDR-Net.

##  1 Introduction

Multi-view stereo (MVS) is a classic and fundamental problem in computer
vision, which utilizes a set of images captured by calibrated cameras from
multi viewpoints to reconstruct the 3D model of a certain scene. It has
various applications in 3D visualization, autonomous driving and robotics [7,
26], etc.

Traditional methods [9, 12, 17, 20] formulate the MVS task as an optimization
problem and utilize hand-crafted metrics to refine this task. Although
traditional methods show great performance on the reconstruction of scenes
with Lambertian surfaces, they still suffer from unreliable matching
correspondences under the condition of illumination changes, low-texture
regions and optical reflections.

With the success of convolutional neural networks (CNNs), many learning-based
methods have been developed to figure out these problems and achieve better
results than traditional methods. Learning-based MVS methods [10, 15, 21, 22,
33, 34, 35] introduce deep CNNs to predict the depth map of each view followed
by a multi-view post-processing for point cloud fusion. These methods infer
the depth value of each pixel by leveraging probability distribution with a
reasonable depth range. Specifically, Gu  _et al_. [10] built a cascaded cost
volume to progressively subpartition the local space and refine the depth
prediction with increasing resolution and accuracy. An essential step in
CasMVSNet is to determine the depth range hypotheses for each stage.
Generally, most methods in a coarse-to-fine manner [10, 33] utilize static
depth range hypotheses through the entire process which is inappropriate for
depth estimation since static hypotheses are unable to adapt to the
uncertainties of previous per-pixel depth predictions. The pixels with high
uncertainties require wider range hypotheses to cover their ground truth depth
values while the pixels with high confidences demand narrower depth ranges to
achieve higher depth sampling rates. Some methods utilize numerical
characteristics of probability distribution to estimate range hypotheses.
Cheng  _et al_. [5] estimated depth ranges by variances and achieved better
results. However, variances are lack of information from nearby pixels since
their reception fields are confined. Moreover, using variances to compute
uncertainties is not suitable for non-Gaussian probability distributions. To
this end, we propose a Dynamic Depth Range Network (DDR-Net) to estimate depth
range hypotheses dynamically, which achieves a significant improvement on
accuracy and completeness for the final point cloud reconstruction.

In this work, we make efforts to estimate more accurate depth range hypotheses
dynamically. We propose a novel cascaded structure with a refined loss
strategy. Range estimation module (REM) is developed to estimate the
uncertainty of a depth range from the previous probability distribution. Range
estimation module (REM) is a shallow 2D CNNs structure. Since the network has
a large reception field, REM is able to gather uncertainty information from
nearby pixels to determine a more accurate depth range. Our proposed DDR-Net
consists of three stages and the range estimation module (REM) is available in
the 1st and 2nd stages because the depth range hypotheses in the 3rd stage are
the final sampling and are directly utilized to produce the final high-
resolution depth map.

To keep corresponding ground truth depth value of each pixel covered in the
hypothesis range of the next stage, we propose a novel loss strategy that
utilizes new sampled depth range hypotheses and a probability distribution of
the former stage to produce a refined depth map. We clamp depth range
hypotheses in the former stage by a new sampled depth range. And the
corresponding probability distribution will also be clamped to obtain a
refined probability distribution. After normalizing the probability
distribution, the refined depth map can be regressed in the $soft$ $argmin$
manner. Through leveraging the refined depth map, the range estimation module
(REM) is able to learn better depth range hypotheses of higher confidences.

Our main contributions are summarized as follows:

  * •

We propose a novel multi-stage MVS framework (DDR-Net) with range estimation
module (REM) to estimate more accurate depth ranges, which dynamically
determines the depth range of every pixel in depth maps and gains more
accurate depth values.

  * •

We design a novel loss strategy to refine our range estimation module (REM),
which further refines our output depth range estimation.

  * •

Extensive experimental results show that our method achieves state-of-the-art
(SOTA) performance on the DTU benchmark and comparable reconstruction results
with other state-of-the-art methods on the Tanks and Temples benchmark.

Figure 2: The full network architecture of our proposed DDR-Net.

##  2 Related Work

Traditional methods. Multi-view stereo has been well studied for decades and a
lot of great methods had been proposed. Traditional MVS methods can be
generally categorized into four types: volumetric methods, patch-based
methods, point-cloud based methods and depth map reconstruction methods.
Volumetric methods [12, 13, 23, 31, 37] construct a bounding volume containing
the scene and divide the volume into small voxel grids. Then these methods
choose the voxels belonging to surface of the scene and filter the unattached
voxels. Patch-based methods [11, 14, 20, 30] consider the surface as a
collection of patches. These approaches first reconstruct the surface in some
areas and then propagate to some low-texture areas. Point-cloud based methods
[8, 17, 18, 29] directly utilize the 3D points to produce results. Depth-map
based methods [2, 9, 25, 27] aim to estimate the depth of the scene and
reconstruct point clouds through various fusion methods. Among these methods,
depth-map based MVS methods have shown more flexibility with low time and GPU
memory consuming which became the major methods utilized in learning-based
frameworks. Although traditional methods achieve remarkable results, they
still suffer from reconstructing illumination-changes, low-texture and
intense-reflection regions.

Learning-based methods. Yao  _et al_. [34] first introduced a differentiable
homography warping to build a variance-based cost volume followed by multi-
scale 3D CNNs for regularization. It achieved incredible results compared with
conventional methods. However, 3D CNNs are generally time and GPU memory
consuming. To allow handling high-resolution images, Yao  _et al_. [35]
utilized a gated recurrent unit (GRU) [6] to replace 3D CNNs in MVSNet to
address the memory-intensive issue. Luo  _et al_. [22] utilized an attention
mechanism to construct and regularize cost volumes. Both perceptual
information and contextual information of the local scene are fused to improve
the matching robustness. Zhang  _et al_. [36] proposed a formulation to
calculate the uncertainties of a cost volume and fuse cost volumes according
to these uncertainties by integrating occluded information in the MVS network.
Xu  _et al_. [32] learned to predict the visibility of every image and present
an anti-noise strategy to refine the visibility network. Recently, a novel
cascade formulation of 3D cost volumes was presented in [5, 10, 33] to reduce
the memory consumption. The fine-grained depth map are refined in a coarse-to-
fine manner. Our methods follow the same coarse-to-fine strategy as these
approaches and propose a novel range estimation module (REM) to estimate a
more accurate depth range.

Depth range hypotheses. Multi-stage multi-view stereo frameworks have been
proposed in [5, 10, 33], which build a cascade cost volume to estimate the
depth map in a coarse-to-fine manner. In these methods, the depth range
hypotheses of the first stage covers the entire depth range of the input
scene. In the following stages, the hypotheses range is narrowed based on the
predicted depth map from the previous stage. CasMVSNet [10] set a reducing
factor of the depth range hypotheses which narrows the depth range manually
and depth ranges of all pixels are identical. CVP-MVSNet [33] found 3D points
that are two pixels away from the its projection along the epipolar line in
both directions on the source view as the depth residual range for depth
refinement. UCSNet [5] leveraged the variance of the depth distribution to
determine the depth range hypotheses in the next stage. Our method is
particularly inspired by this line of work where we propose a range estimation
module (REM) to measure the depth range hypotheses from a probability
distribution combined with a novel designed loss strategy.

##  3 Our Method

In this paper, we propose an effective and efficient multi-view stereo
framework (DDR-Net) that utilizes a novel range estimation module (REM) to
estimate the depth range hypotheses in a coarse-to-fine strategy. Given a
reference image $\mathbf{I}_{1}$ and $N-1$ source images
$\left\\{\mathbf{I}_{i}\right\\}_{i=2}^{N}$, our proposed DDR-Net
progressively regresses a fine-grained depth map using a learned dynamic depth
range. The overall of our proposed network is illustrated in Figure 2. We
first leverage a feature pyramid network (FPN) [19] to extract multi-scale
image features (see Section 3.1) under different image resolutions. Then we
construct multi-scale cost volumes using image features through a homography
warping process and apply 3D CNNs to regularize cost volumes to form a
probability volume (see Section 3.2). A novel range estimation module (REM) is
designed to learn the dynamic depth range by utilizing probability
distributions (see Section 3.3). Finally, we apply supervision to depth output
of all stages and design a loss strategy to constrain our novel REM (see
Section 3.4).

(a) Original image (b) Ground truth

(c) UCSNet [5]

(d) DDR-Net (Ours)

(c) Results from UCSNet [5] (d) Results from DDR-Net

Figure 3: Illustration of detailed depth prediction and dynamic depth range
estimation of a specific example. On the left, we show the RGB image crops
(a), the ground truth depth (b), the predicted depth map (c) from UCSNet [5]
and the predicted depth map (d) from our proposed DDR-Net. On the right, we
show the details of a pixel (red points in the images) with predicted
probability distribution (connected blue dots), the depth value prediction
(red dash line), the ground truth depth (black dash line) and uncertainty
intervals (pink) in the three stages. The top results (e) are from UCSNet [5]
and the bottom results (f) are from our proposed DDR-Net.

###  3.1 Multi-Scale Feature Extraction

Several previous works [21, 34, 35] utilize a shallow network to downsample
original images and extract high-level features with semantic information. Due
to the limit of GPU memory, the cost volume is built upon the final high-level
features and produce the depth map with certain low resolution. In order to
obtain high-resolution depth maps, we utilize a feature pyramid network (FPN)
to extract multi-scale features. As shown in Figure 2, the network only
consists of three stages with several convolutional layers. Given the input
image $\mathbf{I}$ , the final outputs will be $\mathbf{O}_{1}$,
$\mathbf{O}_{2}$, and $\mathbf{O}_{3}$ for building cost volumes. Let the
resolution of input image be $W\times H\times C$, the corresponding
resolutions of output features $\mathbf{O}_{1}$, $\mathbf{O}_{2}$, and
$\mathbf{O}_{3}$ will be $W\times H\times 32$, $\frac{W}{2}\times\
\frac{H}{2}\times 32$, $\frac{W}{4}\times\frac{H}{4}\times 32$, respectively,
where $W$, $H$ and $C$ denote width, height and channel, respectively. Our
multi-scale feature extraction strategy utilizes both high-level semantic
information and low-level structural information to produce high-resolution
depth maps from coarse to fine.

###  3.2 Cost Volume Construction

Similar to previous methods [5, 10], we construct cascade cost volumes in
multi-stages by using the homography warping process in [34] to warp the
features of source views to the reference view. The warping function of the
first stage is expressed as:

$\mathbf{H}_{i}(d)=\mathbf{K}_{i}\mathbf{R}_{i}\bigg{(}\mathbf{I}-\frac{(\mathbf{t}_{1}-\mathbf{t}_{i})\mathbf{n}_{1}^{\top}}{d}\bigg{)}\mathbf{R}_{1}^{\top}\mathbf{K}_{1}^{-1},$
(1)

where $\mathbf{H}_{i}(d)$ refers to the homography between the feature maps of
the $i$-th view and the reference feature map at the depth $d$. Moreover,
$\mathbf{K}_{i}$, $\mathbf{R}_{i}$, and $\mathbf{t}_{i}$ denote the camera
intrinsics, rotations and translations of the $i$-th view, respectively. And
$\mathbf{n}_{1}$ denotes the principle axis of the reference camera. A
variance based metric is then operated to aggregate arbitrary cost volumes of
source views and the reference cost volume to a single one.

In the following stages (the 2nd and the 3rd stages), we follow [10] to set
the homography functions of $(k+1)$-th stage as:

$\mathbf{H}_{i}(d_{k}^{m}\\!+\\!\Delta_{k+1}^{m})=\mathbf{K}_{i}\mathbf{R}_{i}\
\\!\bigg{(}\mathbf{I}-\frac{(\mathbf{t}_{1}\\!-\\!\mathbf{t}_{i})\mathbf{n}_{1}^{\top}}{d_{k}^{m}\\!+\\!\Delta_{k+1}^{m}}\\!\bigg{)}\mathbf{R}_{1}^{\top}\mathbf{K}_{1}^{-1},$
(2)

where $d_{k}^{m}$ denotes the predicted depth of the $m$-th pixel at the
$k$-th stage, and $\Delta_{k+1}^{m}$ is the residual depth of the $m$-th pixel
to be learned at the $(k+1)$-th stage.

After constructing the cost volume, we apply 3D CNNs to regularize the cost
volume to infer the probability volume and get the probability distribution of
the depth. Following previous works, we utilize a 3D U-Net [24] architecture
proposed in [34], which is able to aggregate context information in multi-
scales.

The acquired depth probability volume consists of $D_{k}$ depth probability
maps $\\{\mathbf{P}_{k,j}\\}_{j=1}^{D_{k}}$, where $D_{k}$ is the number of
the depth hypotheses (depth plane) of this stage. Each $\mathbf{P}_{k,j}$ is
associated with a certain depth map $\mathbf{L}_{k,j}$. Then we use a
differentiable $soft$ $argmin$ process to compute the depth value at the pixel
$\mathbf{x}$ as:

$\hat{\mathbf{L}}_{k}(\mathbf{x})=\sum\nolimits_{j=1}^{D_{k}}\left(\mathbf{L}_{k,j}(\mathbf{x})\cdot\mathbf{P}_{k,j}(\mathbf{x})\right).$
(3)

###  3.3 Dynamic Depth Range Estimation

Accurate and appropriate depth range estimation for each pixel on the 2nd and
3rd stages is essential for generating high quality point clouds. To this end,
we propose a novel range estimation module (REM) to estimate the 2nd and the
3rd depth ranges dynamically for each pixel by using the information of a
probability volume in the former stage.

Given a set of probability maps, previous methods only pay attention to the
probability distribution of a single pixel to design the depth range in the
next stage. However, we argue that both the context information and the
probability distribution of the nearby pixels are also significant to
determine the depth ranges of pixels. The illustration of this assumption is
presented in the supplementary details. Also, the information of a probability
distribution is not fully utilized in previous methods which leverage
numerical characteristics to measure the uncertainties. To solve this problem,
we design a range estimation module (REM) to estimate the dynamic depth range
adaptively. In particular, we utilize the previous probability volume and
apply a single network to learn the uncertainties from it. As shown in Figure
2, our REM is composed of several simple 2D CNNs and a sigmoid activation
function. The sigmoid function restricts the values of final output from 0 to
1. Note that the REM is light-weighted but quite effective for the following
depth range hypotheses estimation. With the convolutional network, the
reception field is enlarged and the information of nearby pixels is fully
considered for the following depth estimation. The final output of the network
is an uncertainty map
$\mathbf{C}=\\{\mathbf{C}(\mathbf{x})\\}_{\mathbf{x}\in\mathbf{I}}$ where
$\mathbf{C}(\mathbf{x})$ refers to the uncertainty of the depth distribution
of the pixel $\mathbf{x}$ in the image $\mathbf{I}$. The full network
architecture of our proposed range estimation module (REM) is presented in the
supplementary details.

Given an uncertainty value $\mathbf{C}(\mathbf{x})$ obtained by REM and the
previous depth prediction $\mathbf{L}(\mathbf{x})$ at the pixel $\mathbf{x}$,
we use the following equation to determine the depth range $D(\mathbf{x})$ in
the next stage:

$D(\mathbf{x})=[\mathbf{L}(\mathbf{x})-\lambda\mathbf{C}(\mathbf{x}),\mathbf{L}(\mathbf{x})+\lambda\mathbf{C}(\mathbf{x})],$
(4)

where $\lambda$ is a hyperparameter that determines how large the confidence
interval is. For a pixel $\mathbf{x}$, we uniformly sample the depth values in
the depth range $D(\mathbf{x})$. In this way, we construct dynamic depth
hypotheses for each pixel in the next stage. It is noteworthy that our whole
range estimation module (REM) is light-weighted and time-saving by only adding
a few parameters.

To demonstrate the effectiveness of our module and compare with UCSNet [5], we
show a specific example with a pixel and its estimated uncertainty intervals
around the prediction in Figure 3. Both methods are not able to obtain
satisfactory depth prediction on the first stage. Note that UCSNet fails to
obtain accurate depth prediction because of its inappropriate uncertainty
estimation determined by variance. Its depth range estimation fails to cover
ground truth on the second stage. However, our proposed REM which leverages
probability volume of the former stage and take nearby pixels into
consideration, learns depth range hypotheses with a higher confidence and
adjust to the probability prediction at each stage to achieve better optimized
intervals. In general, our DDR-Net has a better spatial partitioning and
enables to reconstruct the final point clouds with higher completeness and
accuracy.

###  3.4 Loss Strategy

Range estimation module (REM) is developed to dynamically estimate the depth
range of a certain scene for the second and the third stages in our proposed
DDR-Net. However, the estimated depth range may still not accurate enough to
cover ground truth depth values of all pixels. To this end, we design a novel
loss strategy to refine our range estimation module (REM) and produce better
depth ranges with higher confidences.

As shown in Figure 4, given the maximum and minimum depth values of a new
dynamic depth range $D_{\max}^{k+1}$ and $D_{\min}^{k+1}$ in the 2nd stage (or
3rd stage), the probability volume $\mathbf{P}_{k,j}$ and the depth range
hypotheses $\mathbf{L}_{k,j}$ in the former stage, the refined depth range
hypotheses and the probability volume are obtained through the following clamp
operations as:

$\mathbf{L}_{k,j}^{\rm{refined}}(\mathbf{x})\\!=\\!\begin{cases}\mathbf{L}_{k,j}(\mathbf{x}),&D_{\rm{min}}^{k+1}\\!<\\!\mathbf{L}_{k,j}(\mathbf{x})\\!<\\!D_{\rm{max}}^{k+1}\\\
\\\ 0,&\rm{else}\end{cases}$ (5)
$\mathbf{P}_{k,j}^{\rm{refined}}(\mathbf{x})=\begin{cases}\mathbf{P}_{k,j}(\mathbf{x}),&\mathbf{L}_{k,j}^{\rm{refined}}(\mathbf{x})\neq
0\\\ \\\ 0.&\rm{else}\\\ \end{cases}$ (6)

After clamp operations, the refined probability volumes will be normalized to
ensure that the sum probability in a single pixel is one. With refined depth
range hypotheses and refined probability volumes, the final refined depth map
is calculated as:

$\hat{\mathbf{L}}_{k}^{\rm{refined}}(\mathbf{x})=\sum\nolimits_{j=1}^{D_{k}}(\mathbf{L}_{k,j}^{\rm{refined}}(\mathbf{x})\cdot\mathbf{P}_{k,j}^{\rm{refined}}(\mathbf{x})).$
(7)

The resolutions of refined depth maps are the same as the output depth maps in
the 1st and 2nd stages, which are $\frac{W}{4}\times\frac{H}{4}$ and
$\frac{W}{2}\times\frac{H}{2}$, respectively. The ground truth will be
downsampled to the same resolutions.

(a) Original

(b) Clamp

(c) Normalization

Figure 4: Illustration of our novel loss strategy. In (a), we show the
probability distribution of a specific pixel (connected blue dots), depth
prediction (red dash line) and uncertainty intervals (pink areas). Then we
utilize clamp strategy to cut the original probability distribution as
illustrated in (b). Then the refined probability distribution will be
normalized and output the refined depth value (black dash line).

We utilize the smooth L1 loss as our loss function for both initial depth maps
and refined maps. The final loss function is:

$Loss=\sum\nolimits_{k=1}^{3}\alpha_{k}Loss_{k}+\sum\nolimits_{k=1}^{2}\beta_{k}Loss_{k}^{\rm{refined}},$
(8)

where $Loss_{k}$, $Loss_{k}^{\rm{refined}}$, $\alpha_{k}$, and $\beta_{k}$
denote the loss and refined loss of the $k$-th stage and their corresponding
weights, respectively.

We assume that the depth prediction of each stage should have similar
probability distribution even their depth sampling rates progressively
increase. Following this assumption, our loss strategy produces a rough depth
prediction of the new stage in advance by leveraging new estimated depth range
hypotheses. If the estimated depth range does not contain the ground truth
depth value of each pixel, the output refined depth map will have a deviation
compared to the ground truth depth map. In this way, our range estimation
module (REM) can learn an appropriate depth range hypotheses with high
confidences to adjust to the probability distribution. Note that the whole
operation is naturally differentiable and can be implemented as a refine
module in any multi-view stereo framework in a coarse-to-fine manner without
adding any additional parameters.

##  4 Experiments

In this section, we evaluated our proposed DDR-Net in the standard benchmark
of multi-view stereo with a thorough set of experiments to demonstrate
effectiveness and efficiency of our model.

###  4.1 Datasets

DTU dataset [1] is a large-scale MVS dataset with 124 scenes scanned from 49
or 64 views under 7 different lighting conditions. Each scene provides a
reference point cloud and camera parameters of the captured images. Following
CasMVSNet [10], we generated the depth maps for each view in three different
image resolutions as our training ground truth. We used the same training,
validation and evaluation sets as defined in [34].

Tanks and Temples [16] contains realistic scenes in different scales. To
compare with other state-of-the-art methods, we evaluated our model on its
intermediate set which consists of 8 scenes.

Methods | Acc. | Comp. | Overall
---|---|---|---
Camp [3] | 0.835 | 0.554 | 0.695
Furu [8] | 0.613 | 0.941 | 0.777
Tola [28] | 0.342 | 1.190 | 0.766
Gipuma [9] | 0.283 | 0.873 | 0.578
MVSNet [34] | 0.396 | 0.527 | 0.462
R-MVSNet [35] | 0.383 | 0.452 | 0.417
Point-MVSNet [4] | 0.342 | 0.411 | 0.376
CasMVSNet [10] | 0.346 | 0.351 | 0.348
CVP-MVSNet [33] | 0.296 | 0.406 | 0.351
AttMVS [22] | 0.383 | 0.329 | 0.356
Vis-MVSNet [36] | 0.369 | 0.361 | 0.365
UCSNet [5] | 0.338 | 0.349 | 0.344
DDR-Net (Ours) | 0.339 | 0.320 | 0.329
Table 1: Quantitative results of point cloud reconstruction on the DTU dataset
[1]. DDR-Net achieves best completeness and overall score compared with other
state-of-the-art methods

(a) CasMVSNet [10]

(b) DDR-Net (Ours)

(c) Ground Truth

Figure 5: Different results of point cloud reconstruction on the DTU dataset [1]. As shown in orange boxes, our reconstruction generates more complete point cloud with better details, which demonstrates the effectiveness of our method.  Methods | Input Sizes | Depth Map Sizes | Acc. (mm) | Comp. (mm) | Overall (mm) | GPU Memory (MB) | Runtime (s)
---|---|---|---|---|---|---|---
MVSNet [34] |  1600$\times$1184 |  400$\times$288 | 0.396 |  0.527 |  0.462 |  22511 | 2.76
Point-MVSNet [4] |  1600$\times$1184 |  800$\times$576 | 0.342 |  0.411 |  0.376 |  13081 | 3.04
CVP-MVSNet [33] |  1600$\times$1184 |  1600$\times$1184 | 0.296 |  0.406 |  0.351 |  8795 | 1.72
CasMVSNet [10] |  1600$\times$1184 |  1600$\times$1184 | 0.346 |  0.351 |  0.348 |  10153 | 0.89
UCSNet [5] |  1600$\times$1184 |  1600$\times$1184 | 0.338 |  0.349 |  0.344 |  7525 | 0.87
DDR-Net (Ours) |  1600$\times$1184 |  1600$\times$1184 | 0.339 |  0.320 |  0.329 |  7345 | 0.88
Table 2: Comparison of reconstruction quality, GPU memory usage and runtime on the DTU dataset for different input sizes. For the same size of input images, our proposed DDR-Net achieves high quality point cloud reconstruction with smallest GPU memory consumption and reasonable running time. Methods | Mean | Family | Francis | Horse | Lighthouse | M60 | Panther | Playground | Train
---|---|---|---|---|---|---|---|---|---
MVSNet [34] | 43.48 | 55.99 | 28.55 | 25.07 | 50.79 | 53.96 | 50.86 | 47.90 | 34.69
R-MVSNet [35] | 48.40 | 69.96 | 46.65 | 32.59 | 42.95 | 51.88 | 48.80 | 52.00 | 42.38
CVP-MVSNet [33] | 54.03 | 76.50 | 47.74 | 36.34 | 55.12 | 57.28 | 54.28 | 57.43 | 47.54
CasMVSNet [10] | 56.42 | 76.36 | 58.45 | 46.20 | 55.53 | 56.11 | 54.02 | 58.17 | 46.56
UCSNet [5] | 54.83 | 76.09 | 53.16 | 43.03 | 54.00 | 55.60 | 51.49 | 57.38 | 47.89
DDR-Net (Ours) | 54.91 | 76.18 | 53.36 | 43.43 | 55.20 | 55.57 | 52.28 | 56.04 | 47.17
Table 3: Results of point cloud reconstruction on the Tanks and Temples
dataset [16]. Figure 6: The point cloud reconstruction on the Tanks and
Temples dataset [16].

###  4.2 Implementation

Training. We trained our DDR-Net on DTU dataset for all following experiments.
The ground truth depth maps were generated in different image resolutions
($\frac{W}{4}\times\frac{H}{4}$, $\frac{W}{2}\times\frac{H}{2}$ and $W\times
H$) for three stages by down-sampling the original images. We set the
resolution of input images to $640\times 512$ and the number of neighbouring
views to 3. The the same source view selection strategy as in [34] was used in
both training and evaluation. The numbers of depth planes were set to be 48,
32, and 8 for three stages, respectively. The loss weights for three stages
were set as $\alpha_{1}=0.5$, $\alpha_{2}=1.0$, and $\alpha_{3}=2.0$, and the
weights of refined losses were set as $\beta_{1}=3.0$ and $\beta_{2}=0.0$. The
experiments of setting the weights of refined losses are presented in the
supplementary details. The parameter $\lambda$ in Eq. (4) which determines the
confidence interval was set to 1.5 and 0.75 for the 1st and the 2nd stages. We
implemented our work using PyTorch and used the ADAM optimizer to train our
model. The whole network was trained for 16 epochs with a batch size of 2 on
four Nvidia GTX 1080Ti cards. The initial learning rate was set to 0.001 and
was halved iteratively at the 10th, 12th and 14th epochs.

Evaluation. Similar to previous methods, we used 5 neighbouring views for our
testing and set the input image size to be $1600\times 1184$. For the DTU
benchmark, we calculated the accuracy and completeness to demonstrate the
effectiveness of our model. We also calculated the averages of accuracy and
completeness as the overall scores to measure the overall quality of the
reconstruction point clouds. For Tanks and Temples, we used F-score as the
evaluation metric to test our the generalization ability of our model.

Methods | 1st Range | 2nd Range | 2nd Stage Ratio | 3rd Range | 3rd Stage Ratio | Depth Plane
---|---|---|---|---|---|---
CasMVSNet [10] | 508.8mm | 169.72mm | 0.9532 | 21.09mm | 0.8441 | 48,32,8
UCSNet [5] | 508.8mm | 29.46mm | 0.8507 | 10.10mm | 0.7310 | 64,32,8
Ours (REM) | 508.8mm | 145.96mm | 0.9490 | 20.91mm | 0.8391 | 48,32,8
Ours (REM+Loss) | 508.8mm | 139.46mm | 0.9317 | 19.24mm | 0.8435 | 48,32,8
Table 4: Evaluation of the dynamic depth range estimation in the 1st, 2nd and
3rd stages for our proposed DDR-Net with REM and REM+Loss models compared with
CasMVSNet [10] and UCSNet [5].

###  4.3 Results on the DTU Dataset

In this section, we compared our proposed DDR-Net with both traditional
methods and recent learning-based methods. Qualitative comparisons are shown
in Table 1. While Gipuma [9] achieves the best performance in terms of
accuracy, our proposed DDR-Net outperforms other state-of-the-art methods in
both completeness and overscore and ranks the 1st place of the quality of
reconstruction point cloud on the DTU datasets. As shown in Figure 5, our DDR-
Net generates more complete point clouds in the areas with less textures, for
example the boundaries of the church and the white cup covers, which
demonstrate the effectiveness of our DDR-Net.

To demonstrate the efficiency of our method, we further compare our method
with other multi-view stereo frameworks in terms of both GPU memory usage and
runtime. As shown in Table 2, our DDR-Net has a smaller memory usage and
similar testing runtime compared with UCSNet [5] but achieves better point
cloud reconstruction results. In addition, our approach can produce better
results with smaller memory consuming (27.6$\%$ smaller) and lower running
time compared with CasMVSNet [10]. These sufficiently demonstrate the benefit
of our novel range estimation module (REM), in terms of system resource usage.
Our DDR-Net combined with REM obtains high quality point cloud results with
high computation and memory efficiency. More visualized results of the
reconstruction point cloud are presented in the supplementary material.

###  4.4 Results on Tanks and Temples

Here, we evaluated our method on the intermediate set of Tanks and Temples to
test the generalization ability of our model. We used the DTU dataset to train
our network and reconstructed the point cloud in the Tanks and Temples dataset
without any fine-tuning. The source image number is set to $N=5$ for network
inference. Quantitative results are shown in Table 3. Comparing with other
state-of-the art methods that utilize the similar coarse-to-fine strategy, our
DDR-Net achieves a comparable F-score (54.91). In particular, our method
obtains a higher F-score than UCSNet [5], which is the best baseline on the
DTU dataset and is only lower than CasMVSNet [10]. Note that CasMVSNet
leverage more depth planes (160 planes for the first stage) to increase the
depth sampling rate while our method only focuses on improving the estimation
of depth range and utilizes the original depth plane setting (48,32,8) to
construct the point cloud. Qualitative results of our point cloud
reconstruction are shown in Figure 6.

###  4.5 Ablation Study

In this section, we show extensive experimental results to verify the
performance of range estimation module (REM) and the novel loss strategy in
our proposed DDR-Net.

Effectiveness of REM. Our designed range estimation module (REM) aims to learn
the dynamic depth range for each pixel more precisely. We evaluated our
network on the DTU testing dataset. Table 4 shows the average lengths of
dynamic depth ranges and the ratios of the pixels whose estimated depth ranges
cover the ground truth depth values in our proposed DDR-Net. The corresponding
values of both CasMVSNet [10] and UCSNet [5] are also shown in Table 4. To
have a fair comparison, the input image resolution was set to $1152\times 864$
and their original testing parameters were utilized to complete this
experiment. In addition, we observe that our full model (REM+Loss) is able to
learn the dynamic depth range more reasonably. While UCSNet [5] covers short
depth ranges (only 29.46mm in the second stage), its ratios (85.07$\%$ and
73.1$\%$) of covering the ground truth depth values are not sufficient to
output accurate depth map prediction. CasMVSNet [10] obtains the highest
ratios (95.32$\%$ and 84.41$\%$), but the lengths of its depth ranges for all
pixels are not spatially varying. Their depth range hypotheses are not able to
adapt to conditions with different uncertainties. Moreover, their depth ranges
are longer than ours, which means both lower depth-wise sampling rate and
lower accuracy. Our proposed DDR-Net achieves the comparable ratio on the 3rd
stage compared to CasMVSNet (84.35$\%$) but has a shorter range to have
optimized intervals. Note that only the final stage (the 3rd stage) determines
the quality of the final output depth prediction. This justifies that our
estimated uncertainty intervals are of high confidence and also have short
sampling distances to produce the accurate depth map.

Methods | REM | Loss Strategy | Acc. | Comp. | Overall
---|---|---|---|---|---
Baseline [10] |  |  | 0.346 | 0.351 | 0.348
Model A | ✓ |  | 0.333 | 0.341 | 0.337
Model B | ✓ | ✓ | 0.339 | 0.320 | 0.329
Table 5: Ablation study on REM and loss strategy.

The first and second rows in Table 5 shows the results with or without the
range estimation module (REM) compared with the baseline model [10]. Notes
that the range estimation module improves the reconstruction results
significantly in both accuracy and completeness (Acc: from 0.346 to 0.333;
Comp: from 0.351 to 0.341), showing the effectiveness of our range estimation
module.

Effectiveness of loss strategy. To evaluate the effectiveness of our novel
loss strategy, we evaluated our network with or without our loss strategy on
the DTU dataset. The third and fourth rows in Table 4 show the results with or
without our loss strategy. Comparing to the model only with range estimation
module (REM), the model with additional loss strategy can reduce the range of
the third stage from 20.91mm to 19.24mm in average (7.9$\%$ shorter). On the
contrary, the ratio of covering the ground truth depth improves from 0.8391 to
0.8435. This demonstrates that our novel loss strategy can not only increase
the confidence of the estimated depth hypotheses, but also increase the depth
sampling rates to obtain depth maps with higher accuracy. Also, as shown in
Table 5, the model with our novel loss strategy, which is able to estimate
high confidence depth ranges, improves both completeness and overall score
significantly (Comp: from 0.341 to 0.320; Overall: from 0.337 to 0.329). These
results show our novel loss strategy can further improve the reconstruction
results.

##  5 Conclusion

In this paper, we present a DDR-Net which enables to dynamically estimate the
depth range hypotheses. Comparing with previous methods which determine a
depth range of each pixel identically, our proposed range estimation module
(REM) can estimate ranges from global areas and obtain dynamic depth
hypotheses with shorter running time and less memory consumption. Moreover, we
proposed a novel loss strategy to refine REM and boost the reconstruction
results. Experimental results show that our DDR-Net improves accuracy and
completeness of point cloud reconstruction on several standard datasets and
achieves state-of-the-art performance on the DTU benchmark.

## References

  * [1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl.  Large-scale data for multiple-view stereopsis.  International Journal of Computer Vision (IJCV), pages 1–16, 2016\.
  * [2] Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla.  Using multiple hypotheses to improve depth-maps for multi-view stereo.  In European Conference on Computer Vision (ECCV), 2008.
  * [3] Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla.  Using multiple hypotheses to improve depth-maps for multi-view stereo.  In European Conference on Computer Vision (ECCV), 2008.
  * [4] Rui Chen, Songfang Han, Jing Xu, and Hao Su.  Point-based multi-view stereo network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [5] Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su.  Deep stereo using adaptive thin volume representation with uncertainty awareness.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [6] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555, 2014.
  * [7] Yasutaka Furukawa and Carlos Hernández.  Multi-view stereo: A tutorial.  Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015.
  * [8] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multiview stereopsis.  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 32(8):1362–1376, 2009.
  * [9] Silvano Galliani, Katrin Lasinger, and Konrad Schindler.  Massively parallel multiview stereopsis by surface normal diffusion.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  * [10] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [11] Paul Henderson and Vittorio Ferrari.  Learning single-image 3D reconstruction by generative modelling of shape, pose and shading.  International Journal of Computer Vision (IJCV), pages 1–20, 2019\.
  * [12] Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang.  SurfaceNet: An end-to-end 3D neural network for multiview stereopsis.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [13] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  arXiv preprint arXiv:1708.05375, 2017.
  * [14] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3D mesh renderer.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [15] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [16] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: Benchmarking large-scale scene reconstruction.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [17] Maxime Lhuillier and Long Quan.  A quasi-dense approach to surface reconstruction from uncalibrated images.  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 27(3):418–433, 2005.
  * [18] Chen-Hsuan Lin, Chen Kong, and Simon Lucey.  Learning efficient point cloud generation for dense 3D object reconstruction.  In proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [19] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
  * [20] Alex Locher, Michal Perdoch, and Luc Van Gool.  Progressive prioritized multi-view stereo.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [21] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo.  P-MVSNet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [22] Keyang Luo, Tao Guan, Lili Ju, Yuesong Wang, Zhuo Chen, and Yawei Luo.  Attention-aware multi-view stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [23] Stephan R Richter and Stefan Roth.  Matryoshka networks: Predicting 3D geometry via nested shape layers.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
  * [24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015.
  * [25] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [26] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR), volume 1, 2006.
  * [27] Engin Tola, Christoph Strecha, and Pascal Fua.  Efficient large-scale multi-view stereo for ultra high-resolution image sets.  Machine Vision and Applications, 23(5):903–920, 2012.
  * [28] Engin Tola, Christoph Strecha, and Pascal Fua.  Efficient large-scale multi-view stereo for ultra high-resolution image sets.  Machine Vision and Applications, 23(5):903–920, 2012.
  * [29] Jinglu Wang, Bo Sun, and Yan Lu.  MVPNet: Multi-view point regression networks for 3D object reconstruction from a single image.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8949–8956, 2019.
  * [30] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2Mesh: Generating 3D mesh models from single RGB images.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [31] Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum.  Learning shape priors for single-view 3d completion and reconstruction.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [32] Qingshan Xu and Wenbing Tao.  PVSNet: Pixelwise visibility-aware multi-view stereo network.  arXiv preprint arXiv:2007.07714, 2020.
  * [33] Jiayu Yang, Wei Mao, Jose M Alvarez, and Miaomiao Liu.  Cost volume pyramid based depth inference for multi-view stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [34] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  MVSNet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [35] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent MVSNet for high-resolution multi-view stereo depth inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [36] Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang.  Visibility-aware multi-view stereo network.  arXiv preprint arXiv:2008.07928, 2020.
  * [37] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu.  Learning to reconstruct shapes from unseen classes.  arXiv preprint arXiv:1812.11166, 2018.

##  6 Appendix

###  6.1 Overview

In this supplementary details, we first evaluated the effectiveness of our
proposed range estimation module (REM) with additional experiments in Section
6.2. In Section 6.3, more quantitative and qualitative evaluations on our
proposed novel loss strategy are provided. In Section 6.4, more final point
cloud reconstruction results of the DTU dataset [1] are presented. Finally,
some limitations and future work are discussed in Section 6.5.

###  6.2 Additional Experiments on Range Estimation Module (REM)

In this section, we introduce the full network architecture of our proposed
range estimation module (REM) and compare our proposed DDR-Net with other
state-of-the-art methods to demonstrate the effectiveness of our module. All
following experiments were conducted on the DTU testing dataset. To have a
fair comparison, we set the input image resolution to $1152\times 864$ and
utilize original testing parameters of each method to complete following
experiments.

As shown in Table 6, our proposed REM is composed of several simple 2D CNNs
and a sigmoid activation function. The sigmoid function restricts the values
of uncertainty map from 0 to 1. Note that the REM is light-weighted and can be
applied to any multi-view stereo framework in a coarse-to-fine manner. Also,
our REM is quite effective for the depth range hypotheses estimation since its
strong ability to gather probability information from global areas.

We now present the histograms of the estimated dynamic depth ranges of the 3rd
stage of all scans, scan 77 and scan 114 in Figure 7. We also mark the mean
lengths of our DDR-Net and CasMVSNet [10] in the first histograms. The
reference RGB images, our depth predictions and ground truth depth maps of
scan 114 and scan 77 are also provided. Note that the distributions of depth
ranges of all scans meet our expectations: nearly half of all the depth ranges
are of low uncertainties and some portions of the ranges are considered as
high uncertainties. That is reflected by the fact that only boundary of each
object, occluded areas and texture-less regions require longer depth range to
cover their ground truth. Then we show two specific scans to illustrate the
adaptation ability of our REM. Scan 77 is a scan of a white cup and scan 114
is about a monk sculpture. It is observed that scan 77 consists of large
texture-less regions and suffers from optical reflection while the monk
sculpture has rich textures. From the results, we can observe that more pixels
are considered as high uncertainties in scan 77 and more pixels are respected
as high confidences in scan 114. This verifies that our range estimation
module (REM) is able to adapt to different conditions even in texture-less
areas and regions that are suffered from optical reflection.

Layer | Stride | Kernel | Channel | Input
---|---|---|---|---
conv0 | $1\times 1$ | $3\times 3$ | $48/32\rightarrow 16$ | probability volume
conv1 | $1\times 1$ | $3\times 3$ | $16\rightarrow 32$ | conv0
conv2 | $1\times 1$ | $3\times 3$ | $32\rightarrow 32$ | conv1
conv3 | $1\times 1$ | $3\times 3$ | $32\rightarrow 16$ | conv2
conv4 | $1\times 1$ | $3\times 3$ | $16\rightarrow 1$ | conv3
orangesigmoid | — | — | — | conv4
Table 6: The detail architecture of our proposed range estimation module
(REM). Each convolution layer consists of a 2D convolution layer, a batch
normalization layer and a ReLU layer. The sigmoid layer (the colored cell) is
utilized to limit the output values of the uncertainty map to the range [0,
1].

(a) Scan 77 view 3

(b) Depth prediction

(c) Ground Truth

(d) Scan 114 view 8

(e) Depth prediction

(f) Ground Truth

(g) All scans

(h) Scan 77

(i) Scan 114

Figure 7: Histograms of the dynamic depth ranges on the 3rd stage in our
proposed DDR-Net of all scans, scan 77 and scan 114 in the DTU testing
dataset. The $x$-axis denotes the interval length (mm). We mark the mean
values of the lengths of CasMVSNet [10] (black dash line) and our DDR-Net (red
dash line). On the first and second rows, we show RGB image crops, our depth
predictions and ground truth depth maps of scan 77 and scan 114.

(a) RGB image

(b) Ground-truth depth map

(c) RGB image

(d) Ground-truth depth map

(e) UCSNet: depth map

(f) DDR-Net: depth map

(g) UCSNet: depth map

(h) DDR-Net: depth map

(i) Error map

(j) Error map

(k) Error map

(l) Error map

(m) Variance map of stage 1

(n) Uncertainty map of stage 1

(o) Variance map of stage 1

(p) Uncertainty map of stage 1

(q) Variance map of stage 2

(r) Uncertainty map of stage 2

(s) Variance map of stage 2

(t) Uncertainty map of stage 2

Figure 8: Detailed depth predictions, error maps and range estimation of two
specific scenes. In each scene, we show the RGB image crop, the ground depth
map, the corresponding error maps obtained from UCSNet [5] and our DDR-Net.
The variance maps obtained from UCSNet and uncertainty maps produced by DDR-
Net are also provided to illustrate the effectiveness of our proposed range
estimation module (REM).

To further verify the effectiveness of our range estimation module (REM), we
present the original RGB images, the detailed output depth maps, the error
maps, the uncertainty maps obtained by our REM and the variance maps produced
by UCSNet [5] of two scenes in Figure 8. It is observed that our method (DDR-
Net), which leverages 2D CNNs to learn dynamic depth ranges through gathering
probability information of global areas, can estimate more reasonable dynamic
depth ranges than UCSNet [5]. That is because our output uncertainty maps
perform better on the boundary of objects and the texture-less region than the
variance maps. Note that only pixels just close to the edges are considered
with high uncertainty in the variance maps and these pixels are even not next
to each other (margins are highlighted with thin and broken lines). This leads
to the jagged edges of their final depth prediction (as shown in the boundary
of the white cup). On the contrary, our uncertainty maps depict clear and
complete boundaries of each object, which reflect the fact that our range
estimation module takes the probability distribution of each pixel in the
global regions into account. As a result, our final depth outputs are smoother
not only in inner areas but also in the boundary of each object. Also, using
variance to compute uncertainty is unable to produce satisfactory depth
predictions in the texture-less regions (e.g., the white background). This is
reflected by the fact that the information in the texture-less regions is
insufficient to estimate depth ranges with taking only one single pixel into
consideration. In contrast, our REM learns better uncertainties and produce
smoother depth maps with higher accuracies on texture-less areas by utilizing
probability distributions of global areas.

$\beta_{1}$ | $\beta_{2}$ | Acc. | Comp. | Overall
---|---|---|---|---
3.0 | 2.0 | 0.348 | 0.344 | 0.346
3.0 | 1.0 | 0.352 | 0.338 | 0.345
3.0 | 0.5 | 0.340 | 0.324 | 0.332
0.5 | 0.0 | 0.344 | 0.326 | 0.335
1.0 | 0.0 | 0.342 | 0.326 | 0.334
2.0 | 0.0 | 0.346 | 0.322 | 0.334
3.0 | 0.0 | 0.339 | 0.320 | 0.329
Table 7: Detail accuracy, completeness and overall scores with different
settings of weights of refined loss.

(a) Stage 1

(b) Stage 2

(c) Stage 3

Figure 9: Illustration of a failure case with our proposed novel loss strategy
in the stage 2. The clamped probability distribution of the stage 2 is
different from the probability distribution in the stage 3.

###  6.3 Additional Experiments on Loss Strategy

In this section, we discuss the details of our proposed novel loss strategy.
As mentioned before, our proposed novel loss strategy can leverage the
estimated depth range hypotheses to output the refined depth maps in advance.
In our experiments, the most important hyperparameter for loss strategy is the
weight of the refined loss $\beta_{k}$. In order to find the best
hyperparameter settings, we conducted a series of experiments with different
$\beta_{k}$ settings. As shown in Table 7, the best setting of $\beta_{k}$ was
$\beta_{1}$= 3 and $\beta_{2}$= 0 in our experiments. We argue that the novel
loss strategy in the 1st stage is quite enough for accurate depth range
estimation and there might be some bad cases failing to estimate appropriate
refined depth maps in the stage 2. Our assumption for loss strategy is that
the probability distributions in three stages are similar even the depth
sampling rates progressively increase. But we discovered some failure cases
between the stage 2 and the stage 3. As shown in Figure 9, comparing with
initial depth range hypotheses in the stage 1, the new dynamic depth range
hypotheses in the stage 2 are shortened appropriately that the clamped
probability distribution of the stage 1 is basically the same as the
distribution in the stage 2. However, the depth range hypotheses in the stage
3 might be too short that the probability distribution which is clamped by new
estimated range hypotheses is different from the distribution of the stage 3.
If we utilize our novel loss strategy to obtain the refined depth map in
advance in this situation, we may get wrong refined depth prediction.
Therefore, the setting of $\beta_{1}$= 3 and $\beta_{2}$ =0 is good choice for
achieving good performance.

###  6.4 Point Clouds Reconstruction

In this section, we present additional point clouds reconstruction results to
those presented in main paper, as shown in Figure 10. From these results, we
can observe that our proposed DDR-Net can reconstruct various scenes well.

###  6.5 Future Work

Our proposed DDR-Net dynamically infers the depth range hypotheses with a
large reception field for depth prediction. We have evaluated the
effectiveness of estimating dynamic depth range hypotheses. Although DDR-Net
is able to incorporate information from neighborhood pixels to estimate better
depth range hypotheses, some limitations still exist. During sampling with new
depth ranges, the depth sampling intervals and the number of depth planes are
identical which means the whole depth range will be divided equally into
static number of samples.

We argue obtaining adaptive depth sampling intervals and depth planes is
essential for more accurate depth values prediction. The depth range areas
with high probabilities should use small depth sampling intervals for more
depth samples. And the samples of areas with low probabilities should be
reduced which is beneficial for accurate depth estimation. At the same time,
the adaptive depth planes are able to reduce unnecessary depth samples to drop
GPU memory usage and runtime significantly. These ideas probably need a novel
cost volume regularization strategy to learn dynamic depth plane settings for
each pixel.

Figure 10: Details of some reconstructed point clouds on the DTU dataset.