# RobustMVS: Single Domain Generalized Deep Multi-view Stereo

Hongbin Xu Weitao Chen Baigui Sun Xuansong Xie Wenxiong Kang1 1Corresponding
author.Hongbin Xu is with the School of Automation Science and Engineering,
South China University of Technology, Guangzhou, China, 510641. Email:
<EMAIL_ADDRESS>Chen, Baigui Sun, and Xuansong Xie are with the
Damo Academy, Alibaba Group, Hanzhoug, China, 311121.Wenxiong Kang is with the
School of Auomation Science and Engineering, South China University of
Technology, Guangzhou, China, 510641 and Pazhou Lab, Guangzhou, China, 510335.
Email<EMAIL_ADDRESS>
###### Abstract

Despite the impressive performance of Multi-view Stereo (MVS) approaches given
plenty of training samples, the performance degradation when generalizing to
unseen domains has not been clearly explored yet. In this work, we focus on
the domain generalization problem in MVS. To evaluate the generalization
results, we build a novel MVS domain generalization benchmark including
synthetic and real-world datasets. In contrast to conventional domain
generalization benchmarks, we consider a more realistic but challenging
scenario, where only one source domain is available for training. The MVS
problem can be analogized back to the feature matching task, and maintaining
robust feature consistency among views is an important factor for improving
generalization performance. To address the domain generalization problem in
MVS, we propose a novel MVS framework, namely RobustMVS111The code is released
at: https://github.com/ToughStoneX/Robust-MVS.. A Depth-Clustering-guided
Whitening (DCW) loss is further introduced to preserve the feature consistency
among different views, which decorrelates multi-view features from viewpoint-
specific style information based on geometric priors from depth maps. The
experimental results further show that our method achieves superior
performance on the domain generalization benchmark222The MVS evaluation
benchmark is released at:
https://github.com/ToughStoneX/MVS_Evaluation_Benchmark.

###### Index Terms:

Multi-view Stereo, Domain Generalization, Deep Learning, 3D Reconstruction,
Computer Vision

##  I Introduction

Multi-view Stereo (MVS) is a long-standing elementary problem in computer
vision, which aims to reconstruct 3D scenarios from multi-view images. In
recent years, immense progress in this field has been witnessed since the
advent of learning-based MVS. Typically, a MVS network [1] extracts multi-view
image features with a shared CNN first and constructs cost volume via plane
sweep algorithm [2] where source images are reprojected to the reference view
frustum. Afterward, this cost volume is further regularized by 3D CNN [3, 4,
5] or recurrent neural networks [6]. Despite the amazing reconstruction
performance and efficiency of learning-based MVS methods, the hunger for
large-scale 3D annotations limits the application in realistic scenarios.
Consequently, some unsupervised MVS methods [7, 8, 9] are further proposed to
handle this issue without accessing ground truth.

In previous state-of-the-art methods, much effort has been devoted to making
more accurate and complete reconstruction results, but relatively little has
been put into understanding the domain generalization problem in MVS. In this
paper, we aim to investigate the issue of generalizing to unseen domains in
multi-view stereo (MVS), particularly the variations in illuminance among
different domains. Specifically, we focus on a realistic and challenging
scenario where there is only one source domain available but the goal is to
generalize to target domains without any fine-tuning or adaptation, as
illustrated in Fig. 1. The major difficulties lie in the large domain
differences (visualized via MMD distance in Fig. 1) among different MVS
datasets, including factors like variations of color, illumination, and
texture. An interesting phenomenon can be found that the domain gap of BL
$\rightarrow$ TT is much smaller than the one of DT $\rightarrow$ TT, which
accounts for the better performance on TT test set when using BL as training
set compared with DT, verifying the experiments in previous works[10, 5].

Figure 1: Illustration of the proposed MVS domain generalization task. A
single source domain is selected for training, and further tested on target
domains without finetuning. The MMD Distances among datasets are visualized on
the right hand. MVS datasets: DTU (DT) [11], BlendedMVS (BL) [12], GTASFM (GS)
[13], PASMVS (PA) [14], Tanks&Temples (TT) [15]. Figure 2: Visualization of
the activation map of CasMVSNet [3] via Grad-Cam [16]. GS $\rightarrow$ DT
means training on domain GS and generalizing to unseen domain DT. DT
$\rightarrow$ DT means training and testing on the same domain DT.

To handle the domain discrepancy, a common approach is to adapt the source
domain distribution to the target domain distribution via domain adaptation
(DA) techniques [17, 18, 19, 20]. Whereas, it requires access to unlabeled
samples in target domains, which limits their applicability in our MVS
generalization task. A better reference is the domain generalization (DG)
technique [21, 22, 23], which aims to learn domain-invariant representations.
Furthermore, recent advances in the stereo matching task [24, 25, 26, 27, 28,
29, 30] have been developed to handle the domain generalization issue via
conducting feature-level alignment for obtaining domain-invariant features.
The stereo matching task can be viewed as a simplified case of MVS, which
suffers less from view-point variations and shares more common regions.
Instead of direct migration to the MVS case, we would like to take a step back
first and revisit the bottleneck of generalization to unseen MVS domains.

To provide some inductive inspirations, we conduct a toy experiment that
visualizes the feature activation map of an MVS network via modified Grad-CAM
[16], as shown in Fig. 2. The first row shows the original multi-view images,
and the following two rows respectively visualize the activation map when
generalizing to unseen (GS $\rightarrow$ DT) and seen (DT $\rightarrow$ DT)
domains. The non-activated regions (the blue circled area in Fig. 2) reveal
the ignorance of MVS network during generalization. Consequently, the
bottleneck of domain generalization in MVS can be resorted to the robustness
of cross-view matching features across domain-specific conditions of
illumination, color, and reflectance.

In this paper, we propose a novel MVS framework, RobustMVS, with minimum
modification to the backbone MVS network. To handle the aforementioned issue
that disturbs the feature consistency, we aim to suppress the domain-specific
features and reserve the domain-invariant ones. As a series of studies [31,
32, 33, 34, 35] claims, feature covariance implies domain-specific style such
as color and illumination. Based on this claim, the whitening transformation
[27, 36] is a technique designed for removing feature correlation and thus
enhancing the invariance to style variation. However, simply adopting the
whitening transformation may not be applicable for MVS, because the whitening
transformation decouples the feature style towards the whole image while MVS
requires robust feature consistency among views in the local regions. Hence,
we introduce a novel Depth-Clustering-guided Whitening (DCW) loss in the
RobustMVS framework. With minor modifications to the MVS network, the last BN
layer in each of the first 3 convolutional blocks is replaced with IN layer
for normalization, as suggested by IBN-Net [37]. The whitening transformation
is designed as a regularization loss without adding extra modules to the
original network. Our DCW loss firstly clusters the merged sparse point clouds
generated from multi-view depth annotations to find out common regions which
are spatially close. Then the clusters are projected to each view which
determines the corresponding local regions in different views. By segmenting
the local corresponding region (cluster) out, we warp the cross-view features
to the same viewpoint via homography warping and calculate the feature
covariance matrices on these correlated features. Then the feature whitening
constraints are applied to enhance the domain-invariant features in these
local correlated regions. Our experiments on MVS domain generalization task
demonstrate the superior performance of the proposed framework.

The main contributions include the following:

  * •

We systematically investigate the domain generalization problem in MVS and
propose a new domain generalized benchmark across multiple datasets in MVS.
Different from the original ones, the evaluation protocols (BL/GS/GS) are
specially customized for measuring the performance of 3D reconstruction.

  * •

We propose a novel domain generalized MVS framework, RobustMVS. By applying
minor modifications to the MVS network and training with the proposed Depth-
Clustering-guided Whitening (DCW) loss, the RobustMVS framework can improve
the generalization ability with negligible computational cost.

  * •

The domain generalized evaluation results show the superiority of our approach
over existing approaches in both a qualitative and quantitative manner.

##  II Related Work

In this section, we briefly introduce the learning-based Multi-view Stereo
(MVS) methods that are highly related to our paper. In addition, we also
review some related work of Domain Generalization methods to explore the
single domain generalization classification task and their relationship with
our work.

###  II-A Learning-based MVS

With the blessing of large-scale MVS datasets like DTU [11] and BlendedMVS
[12], learning-based MVS methods are powered by deep convolutional networks
[1], recurrent networks [6], and Transformers [10], achieving much better
reconstruction accuracy and completeness than traditional MVS methods [38, 39,
40]. The pioneering work of MVSNet [1] projects the feature maps of multi-view
images extracted by CNN to the same reference view frustum, and builds a 3D
cost volume to model the matching cost on each hypothetical depth plane. 3D
CNN is then used for cost aggregation, and the output per-view depth maps are
fused to construct a 3D point cloud. Due to the high memory and computation
cost of MVSNet, lots of efforts have been proposed which can be divided into
recurrent MVS networks and coarse-to-fine MVS networks. The recurrent methods
[41, 6, 42] use the Recurrent Neural Network (RNN) to regularize the cost
volume as an alternative to CNNs. In this way, the computational efficiency is
sacrificed in exchange for reducing memory cost. When the coarse-to-fine
methods [43, 4, 44, 3, 45] are adopted, the cost volume at the coarsest stage
is constructed by uniformly sampling depth planes [46] of all depth
hypotheses. Then the depth map is refined iteratively by building a more
compact cost volume according to the estimated depth in previous stage as an
initialization. CVP-MVSNet [43] proposes to adaptively determine the local
depth search ranges by calculating the mean depth interval corresponding to
0.5 pixel distance on the epipolar line of the closest source images. UCSNet
[4] utilizes the variance of the depth probability volume and depth residual
to determine the pixel-wise depth interval when sampling depth hypotheses.
UGNet [44] enables the MVS network to perceive the uncertainty, that is
further used to guide the depth hypothesis sampling and build more compact
cost volumes without redundancy. CasMVSNet [3] relies on the hyperparameters
obtained from practice to determine the depth interval in coarse-to-fine cost
volumes. Furthermore, some recent works [47, 10] attempt to embed MVS network
with Transformer [48] backbones, achieving impressive improvement compared
with previous methods. Another recent strand of work is unsupervised MVS [7,
8, 9, 49, 50], which designs self-supervised loss based on photometric
consistency [51, 52] to train the MVS networks, producing competitive
performance compared to supervised methods. Despite the promising results of
learning-based MVS methods, the generalization problem in MVS has not been
systematically explored yet. Though the generalization experiments to
Tanks&Temples are widely used in previous methods, the experimental settings
might vary because some methods [1, 6] adopt DTU as the source dataset while
others utilize BlendedMVS [53] or multiple datasets [10]. As Fig. 1 shows, the
distribution gap of BlendedMVS is much smaller than DTU, which might affect
the final generalization performance when different source data settings are
adopted. The exact influence of distribution among different MVS datasets has
not been systematically investigated yet. More challenging but worthwhile, we
introduce a novel domain generalization problem in MVS and aim to handle it in
this paper.

###  II-B Domain Generalization in Stereo Vision

Domain generalization [21, 22, 23] aims to learn from multiple source domains
but generalize to unseen target domains. These techniques have also been
extended to the era of stereo matching [24, 25, 26, 27], which is a special
case of MVS with horizontally aligned camera poses. DSMNet [25] adopts a
domain normalization layer to reduce shifts of domain-specific styles in
images, and CFNet [26] proposes a fused cost volume representation that is
robust to domain variations. FCStereo [27] involves selective whitening
constraints to stereo matching to suppress the domain-sensitive features. The
domain generalized stereo matching methods provide a reliable reference, and
we extend the state-of-the-art methods to MVS network for comparison in
Section V.

##  III Preliminary

In this section, we briefly introduce the concept of feature whitening
transformation operation and its application to domain generalized tasks like
Semantic segmentation and Stereo Matching in Section III-A. Then we discuss
the limitation of the Whitening loss towards our MVS task in Section III-B.

###  III-A Whitening Transformation and Whitening Loss

Denote that $\mathbf{F}\in\mathbb{R}^{C\times{N_{HW}}}$ is a flattened feature
map, whose number of channels is $C$ and $N_{HW}=H*W$. $H$ and $W$
respectively represent the height and width. As previous studies [31, 32, 33,
34, 35] claim, feature covariance matrix
$\mathbf{F}\cdot\mathbf{F}^{T}\in\mathbb{R}^{C\times C}$ contains the visual
style information of the whole image. Based on this intuition, whitening
transformation (WT) is a linear transformation that aims to make the variance
term of each channel in the feature map equal to one and the covariance term
among different channels equal to zero. More concretely, it aims to find a
whitened feature map $\tilde{\mathbf{F}}\in\mathbb{R}^{C\times{N_{HW}}}$ that
satisfies
$\tilde{\mathbf{F}}\cdot\tilde{\mathbf{F}}^{T}={N_{HW}}\cdot\mathbf{I}\in\mathbb{R}^{C\times
C}$, where $\mathbf{I}$ is an identity matrix.

The mean vector of the feature map can be calculated as:

$\bm{\mu}=\frac{1}{{N_{HW}}}\mathbf{F}\cdot\mathbf{1},\mathbf{1}\in\mathbb{R}^{{N_{HW}}\times
1}$ (1)

Then the covariance matrix can be computed as:

$\mathbf{\Sigma}=\frac{1}{{N_{HW}}}(\mathbf{F}-\bm{\mu}\cdot\mathbf{1}^{T})(\mathbf{F}-\bm{\mu}\cdot\mathbf{1}^{T})^{T},\mathbf{1}\in\mathbb{R}^{1\times{N_{HW}}}$
(2)

Afterwards, the solution of whitened feature map is:

$\tilde{\mathbf{F}}=\mathbf{\Sigma}^{-\frac{1}{2}}(\mathbf{F}-\bm{\mu}\cdot\mathbf{1}^{T}),\mathbf{1}\in\mathbb{R}^{1\times{N_{HW}}}$
(3)

The covariance matrix $\mathbf{\Sigma}$ can be eigen-decomposed to
$\mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{T}$, where
$\mathbf{Q}\in\mathbb{R}^{C\times C}$ is the orthogonal matrix of
eigenvectors, and $\mathbf{\Lambda}\in\mathbb{R}^{C\times C}$ is the diagonal
matrix that contains eigenvalues. The inverse square root of the covariance
matrix can be calculated with
$\mathbf{\Sigma}^{-\frac{1}{2}}=\mathbf{Q}\mathbf{\Lambda}^{-\frac{1}{2}}\mathbf{Q}^{T}$,
substituting the corresponding term in Eq. 3.

Since computing whitening transformation matrix analytically during inference
is computationally expensive, GDWCT [54] converts the whitening constraints to
a regularization loss on feature maps:

$L_{\text{whiten}}=\mathbb{E}[\|\mathbf{\Sigma}-\mathbf{I}\|_{1}]$ (4)

The whitening loss $L_{\text{whiten}}$ and the costful computation process of
calculating eigendecomposition are only used during training, while no extra
computation is conducted during inference.

###  III-B Limitations of Whitening Loss in MVS

With the nature of removing style information [55], the whitening constraints
can be appended to the domain generalization task for learning domain-
invariant representations. Whereas the domain-specific and domain-invariant
features are simultaneously encoded in the covariance matrix of feature maps,
they may be diminished by whitening loss at the same time. Some extensions
[36, 27] handle this by selecting the domain-specific and domain-invariant
terms of the covariance matrix via its sensitivity to changes like photometric
transformations. However, these whitening losses are operated on the whole
image to regularize the style difference among all pixels in the whole image.
It is in conflict with the core target of MVS, which concentrates on the
correspondence among pixels in local regions rather than the whole image. As
the discussion in Section I of Fig. 2 reveals, the robustness of local
corresponding features among views towards changes is the bottleneck when
generalizing to unseen domains. Hence, instead of concentrating on the whole
image with whitening loss, we propose a novel Depth-Clustering-guided
Whitening (DCW) loss that applies whitening loss to the local corresponding
regions of all views.

##  IV Method

In this section, we design a novel RobustMVS framework to handle the domain
generalized problem in MVS. In Section IV-A, we investigate the potential
challenges of the domain generalized problem in MVS. Then, we introduce the
adopted backbone network of the MVS framework in Section IV-B. In Section
IV-C, we introduce the modification of MVS networks to support the feature
whitening operations. Afterwards, we introduce the Depth-Clustering-guided
Whitening (DCW) loss of this work which aims to conduct feature whitening
selectively on the local corresponding regions among different views.

Figure 3: Illustration of our proposed RobustMVS framework. Aug. means random
data augmentation. The blue dashed box represents the clustering process with
the depth prior. This process can extract local corresponding regions by
clustering, and the clusters are further fed to the proposed Depth-Clustering
Guided Whitening loss shown in the purple dashed box. The orange dashed box at
the bottom represents the basic architecture of the adopted MVS network, in
which the first 3 BN convolution blocks are replaced with the IN-based
convolution blocks.

###  IV-A Problem Statement and Notation

We first introduce the notations and definition of the domain generalization
problem in MVS. The goal of our MVS domain generalization task is to learn a
model that is trained only using data from a source domain that can generalize
well to the unseen target domain. In the source domain, given a pair of multi-
view images with $N$ views, the image on reference view is noted as
$I_{\text{ref}}=I_{1}$, and the source view image is denoted as
$I_{\text{src}}=\\{I_{v}\\}_{n=2}^{N}$. In the same way, the intrinsic and
extrinsic matrix are written as: $K_{\text{ref}}$/$K_{\text{src}}$,
$T_{\text{ref}}$/$T_{\text{src}}$. Accordingly, the ground truth depth map is
written as $D_{\text{ref}}$/$D_{\text{src}}$.

###  IV-B Backbone

As shown in the third row of Fig. 3, the backbone MVS network of RobustMVS is
a multi-stage MVS network that follows a coarse-to-fine pipeline as discussed
in Section II. In default, we choose the foundational CasMVSNet [3] as the
backbone. On each view, the output feature map of the feature extraction
network is projected to the reference view via homography warping. Based on
the given depth hypothesis either from initial settings or from the prediction
of the previous stage, the cost volume is built to measure the similarity of
corresponding points among views. Then the cost volume is fed to a 3D CNN in
the cost regularization process, and finally outputs the predicted depth.

###  IV-C Convolutional Block with Normalization

The optimization of the aforementioned whitening loss in Eq. 4 can be
separated into two parts: optimization on the diagonal and off-diagonal of the
covariance matrix. Denote that $F_{i}$ is channel $i$ of feature map $F$. For
the diagonal constraints, its loss can be written as:
$\|\mathbf{\Sigma}-1\|_{1}=\|\frac{F_{i}F_{i}^{T}}{{N_{HW}}}-1\|_{1}$. For the
off-diagonal constraints, its loss can be written as:
$\|\mathbf{\Sigma}\|_{1}=\|\frac{F_{i}F_{j}^{T}}{{N_{HW}}}\|_{1}$. The former
constraint forces the scale of channel $F_{i}$ to be $\frac{1}{{N_{HW}}}$,
while the latter constraint forces the target to be zero. This conflict may
confuse the training process. Hence, a simple solution to handle this issue is
to standardize the feature map with Instance Normalization [56] to ensure a
fixed scale for each channel $F_{i}$:

$\mathbf{F}_{s}=(\text{diag}(\mathbf{\Sigma}))^{-\frac{1}{2}}\odot(\mathbf{F}-\bm{\mu}\cdot\mathbf{1}^{T}),\mathbf{1}\in\mathbb{R}^{1\times{N_{HW}}}$
(5)

where $\odot$ is an element-wise multiplication. Then the normalized feature
map can be used to compute the covariance matrix Eq. 2 and the whitening loss
via Eq. 4

The summarized principle is that the whitening loss should be computed on the
standardized output of an Instance Normalization layer. With this in mind, we
replace the Batch Normalization layer with the Instance Normalization layer
when the output feature map is used to calculate the whitening loss, as shown
in Fig. 3.

###  IV-D Depth-Clustering-guided Whitening Loss

Since the core of MVS is the feature-matching task, which concentrates on the
correspondence in local regions. It is inappropriate to directly append
whitening constraints to features in the whole image, because it may diminish
specific style patterns which is beneficial for feature consistency in local
regions. Hence, we propose a novel Depth-Clustering-guided Whitening (DCW)
loss to conduct feature whitening selectively on the local corresponding
regions among different views. In this section, we first introduce the
procedure of segmenting the local correspondence via clustering with depth
priors in Sec. IV-D1, and then expound the feature whitening on cross-view
corresponding features by homography warping in Sec. IV-D2.

####  IV-D1 Clustering with Depth Priors

As shown in the first row of Fig. 3, the ground truth depth maps of reference
view ($D_{\text{ref}}\in\mathbb{R}^{H\times W}$) and source view
($D_{\text{src}}\in\mathbb{R}^{H\times W}$) are first projected back into 3D
space.

$P_{\text{ref}}^{i}=T_{\text{ref}}^{-1}K_{\text{ref}}^{-1}D_{\text{ref}}(p_{\text{ref}}^{i})p_{\text{ref}}^{i}$
(6)
$P_{\text{src}}^{j}=T_{\text{src}}^{-1}K_{\text{src}}^{-1}D_{\text{src}}(p_{\text{src}}^{j})p_{\text{src}}^{j}$
(7)

where $i$ and $j$ are respectively the index of valid points in
$D_{\text{ref}}$ and $D_{\text{src}}$, and $p_{\text{ref}}$ and
$p_{\text{src}}$ represent the 2D image coordinates. Denote that the number of
valid pixels in the reference view is $M_{\text{ref}}$ and the one in the
source view is $M_{\text{src}}$. The projected point cloud on reference view
is $P_{\text{ref}}\in\mathbb{R}^{M_{\text{ref}}\times 3}$, and the point cloud
on source view is $P_{\text{src}}\in\mathbb{R}^{M_{\text{src}}\times 3}$.

Then the point clouds $P_{\text{ref}}$ and $P_{\text{src}}$ are concatenated
together to get a fused point cloud
$P_{\text{all}}\in\mathbb{R}^{(M_{\text{ref}}+M_{\text{src}})\times 3}$.
Afterward, K-Means [57] is adopted to cluster the spatially neighboring
regions of the fused point cloud $P_{\text{all}}$. The number of clusters is
defined as $K_{\text{clu}}$, and the output clustered point cloud is
$S_{\text{all}}\in\mathbb{R}^{(M_{\text{ref}}+M_{\text{src}})\times{K_{\text{clu}}}}$.
The $K_{\text{clu}}$ channels of point cloud $S_{\text{all}}$ use one-hot
encoding to represent the $K_{\text{clu}}$ clusters. Each of the
$K_{\text{clu}}$ clusters in point cloud $S_{\text{all}}$ represents the
spatially neighboring regions that bind the matching consistency of local
regions between different views. Afterward, point cloud $S_{\text{all}}$ can
be split back into clusters on reference view and source view:
$S_{\text{ref}}\in\mathbb{R}^{M_{\text{ref}}\times{K_{\text{clu}}}}$ and
$S_{\text{src}}\in\mathbb{R}^{M_{\text{src}}\times{K_{\text{clu}}}}$. Note
that the $K_{\text{clu}}$ clusters in $S_{\text{ref}}$ and $S_{\text{src}}$
are corresponding regions that inherently model the matching consistency among
views.

The clustered point clouds can then be projected into the 2D image space:

$\tilde{S}_{\text{ref}}(p_{\text{ref}}^{i})=S_{\text{ref}}(P_{\text{ref}}^{i}),p_{\text{ref}}^{i}=\pi(K_{\text{ref}}T_{\text{ref}}P_{\text{ref}}^{i})$
(8)
$\tilde{S}_{\text{src}}(p_{\text{src}}^{j})=S_{\text{src}}(P_{\text{src}}^{j}),p_{\text{src}}^{j}=\pi(K_{\text{src}}T_{\text{src}}P_{\text{src}}^{j})$
(9)

where $\pi([x,y,z]^{T})=[x/z,y/z,1]$, $i$ and $j$ are respectively the index
of valid points which can be projected into 2D image.
$\tilde{S}_{\text{ref}}\in\mathbb{R}^{H\times W\times{K_{\text{clu}}}}$ and
$\tilde{S}_{\text{src}}\in\mathbb{R}^{H\times W\times{K_{\text{clu}}}}$ are 2D
clustered segmentation maps that share common clusters among different views.

####  IV-D2 Homography Whitening

Denote that $\mathbf{F}_{\text{ref}}^{l}\in\mathbb{R}^{{N_{HW}}\times C}$ and
$\mathbf{F}_{\text{src}}^{l}\in\mathbb{R}^{{N_{HW}}\times C}$ are the output
feature maps on the $l$-th layer, which is normalized with Instance
Normalization following Eq. 5. As shown in Fig. 3, each feature map
$\mathbf{F}_{s}^{l}$ is filtered with each of the $K_{\text{clu}}$ clusters to
find locally consistent regions:

$\tilde{\mathbf{F}}_{\text{ref}}^{l,k}=\mathbf{F}_{\text{ref}}^{l}\odot\mathbb{1}(\tilde{S}_{\text{ref}}^{k}=1)$
(10)
$\tilde{\mathbf{F}}_{\text{src}}^{l,k}=\mathbf{F}_{\text{src}}^{l}\odot\mathbb{1}(\tilde{S}_{\text{src}}^{k}=1)$
(11)

where $\odot$ is an element-wise multiplication.
$\tilde{\mathbf{F}}_{\text{ref}}^{l,k}\in\mathbb{R}^{M_{ref}^{k}\times C}$ and
$\tilde{\mathbf{F}}_{\text{src}}^{l,k}\in\mathbb{R}^{M_{src}^{k}\times C}$ are
the pixel-wise features of cluster $k$ on the reference view and the source
view. $M_{ref}^{k}$ and $M_{src}^{k}$ are the numbers of valid pixels in
cluster $K$ on reference view and source view.

Since $M_{ref}^{k}$ and $M_{src}^{k}$ are probably unequal, they can not be
directly used to calculate the covariance matrix that has to be a square
matrix. Hence, we reproject $\tilde{\mathbf{F}}_{\text{ref}}^{l,k}$ to the
source view and $\tilde{\mathbf{F}}_{\text{src}}^{l,k}$ to the reference view
via homography warping:

$\tilde{\mathbf{F}}_{\text{ref}\rightarrow\text{src}}^{l,k}[p_{\text{src}}]=\tilde{\mathbf{F}}_{\text{ref}}^{l,k}[\pi(K_{\text{ref}}T_{\text{ref}}T_{\text{src}}^{-1}K_{\text{src}}^{-1}p_{\text{src}})]$
(12)
$\tilde{\mathbf{F}}_{\text{src}\rightarrow\text{ref}}^{l,k}[p_{\text{ref}}]=\tilde{\mathbf{F}}_{\text{src}}^{l,k}[\pi(K_{\text{src}}T_{\text{src}}T_{\text{ref}}^{-1}K_{\text{ref}}^{-1}p_{\text{ref}})]$
(13)

where
$\tilde{\mathbf{F}}_{\text{src}\rightarrow\text{ref}}^{l,k}\in\mathbb{R}^{M_{ref}^{k}\times
C}$ and
$\tilde{\mathbf{F}}_{\text{ref}\rightarrow\text{src}}^{l,k}\in\mathbb{R}^{M_{src}^{k}\times
C}$ are the warped features via bilinear sampling to ensure the backward
propagation of gradient during training.

Then we can calculate the covariance matrix:

$\mathbf{\Sigma}_{\text{ref}\rightarrow\text{src}}^{l,k}=(\tilde{\mathbf{F}}_{\text{src}}^{l,k})^{T}\tilde{\mathbf{F}}_{\text{ref}\rightarrow\text{src}}^{l,k}$
(14)
$\mathbf{\Sigma}_{\text{src}\rightarrow\text{ref}}^{l,k}=(\tilde{\mathbf{F}}_{\text{ref}}^{l,k})^{T}\tilde{\mathbf{F}}_{\text{src}\rightarrow\text{ref}}^{l,k}$
(15)

where the covariance matrix
$\mathbf{\Sigma}_{\text{ref}\rightarrow\text{src}}^{l,k}$ and
$\mathbf{\Sigma}_{\text{src}\rightarrow\text{ref}}^{l,k}$ have the resolution
of $C\times C$.

Note that the original input of the reference image and source image are
randomly augmented with photometric transformations as shown in the left-
bottom of Fig. 3. Here, the covariance matrix
$\mathbf{\Sigma}_{\text{ref}\rightarrow\text{src}}^{l,k}$ and
$\mathbf{\Sigma}_{\text{src}\rightarrow\text{ref}}^{l,k}$ models the style
information on the local region of the same cluster $k$. With the random data
augmentation, we would like to find out which elements of the covariance
matrix is sensitive to changes, and suppress them with whitening loss.
Formally, the variance matrix between these covariance matrices is calculated
as follows:

$\bm{\mu}_{\mathbf{\Sigma}}=\frac{\mathbf{\Sigma}_{\text{ref}\rightarrow\text{src}}^{l,k}+\mathbf{\Sigma}_{\text{src}\rightarrow\text{ref}}^{l,k}}{2}$
(16)
$\mathbf{V}=\frac{(\mathbf{\Sigma}_{\text{ref}\rightarrow\text{src}}^{l,k}-\bm{\mu}_{\mathbf{\Sigma}})^{2}+(\mathbf{\Sigma}_{\text{src}\rightarrow\text{ref}}^{l,k}-\bm{\mu}_{\mathbf{\Sigma}})^{2}}{2}$
(17)

where $\mathbf{V}$ contains the variance of each covariance element across
random data augmentation towards consistent regions on different views. Note
that The random data augmentation transformations include the color jittering
and gamma correction. Color jittering generates random variations of
brightness, contrast, and saturation. The brightness and contrast factor are
set to 0.7, and the one of saturation is set to 0.2. Gamma correction randomly
samples gamma values within a range of 0.5-2.0 to augment the images.

As presented in the figure, we then filter out the sensitive element from
$\mathbf{V}$ with a threshold $\tau$ using the indicator function:
$\mathbb{1}(\mathbf{V}>\tau)$. $\tau$ is an adaptive threshold calculated
following [36] (Details are discussed in the appendix). Furthermore, since the
covariance matrix is a symmetric matrix, only the upper triangle elements are
useful for the whitening loss. Hence, we can calculate the mask given the
upper triangle matrix $\text{triu}(\bm{1})$:

$\mathbf{O}=\mathbb{1}(\mathbf{V}>\tau)\odot\text{triu}(\bm{1}),\bm{1}\in\mathbb{R}^{C\times
C}$ (18)

Finally, the DCW loss is defined as:

$L_{\text{DCW}}^{l,k}=\mathbb{E}[\|\Sigma_{\text{ref}\rightarrow\text{src}}^{l,k}\odot\mathbf{O}-\epsilon\|_{1}+\|\Sigma_{\text{src}\rightarrow\text{ref}}^{l,k}\odot\mathbf{O}-\epsilon\|_{1}]$
(19)

where $0<\epsilon<<1$ is set as a threshold aimming to relax the constraints
of whitening loss during optimization. $\epsilon$ is added to avoid the
unexpected failures (NaN values) caused by whitening loss.

####  IV-D3 Overall loss

The final training loss is a weighted sum of depth estimation loss in MVS and
the proposed DCW loss:

$L=L_{\text{depth}}+\frac{\lambda}{(N-1)LK}\sum_{n=1}^{N-1}\sum_{l=1}^{L}\sum_{k=1}^{{K_{\text{clu}}}}L_{\text{DCW}}^{l,k}$
(20)

where $L_{\text{depth}}$ is the commonly applied smooth huber loss following
[3]. $\lambda$ is the weight of DCW loss and is set to $1$ in default. The $N$
views of multi-view images are separated in $N-1$ paired samples comprised of
reference view ($n=1$) and source view ($n=2,...,N$). $L$ layers output
standardized features to compute whitening loss, and $K_{\text{clu}}$ clusters
are used as shown in Fig. 3.

##  V Experiment

###  V-A Datasets

DTU (DT) [11] is an indoor MVS dataset. Each scene is taken from 49 positions
and the image resolution is $1600\times 1200$. 7 different light conditions
are included in each scene. Following the separation of MVSNet [1], 78 scenes
are included in the training set, 17 scenes are used in the validation set,
and 21 scenes are used in the test set.

BlendedMVS (BL) [12] is a large-scale MVS dataset, which contains various
scenes, including architectures, sculptures, and small objects. Following
their official split333https://github.com/YoYo000/BlendedMVS, 106 scenes are
used in the training set and 7 scenes are used in the evaluation set.

GTASFM (GS) [13] is a synthetic dataset collected from GTA-V, a famous video
game with realistic scenarios. Following the official split, 200 scenes are
included in the training set, and 19 scenes are included in the test set. The
original images provided in GS are a sequence of images extracted from a
video, paired with camera parameters and ground truth depth maps. To support
the format of MVS networks, these image sequences are firstly processed with
COLMAP and the neighboring images are selected based on the similarity score
of matching pixels among views. The format is saved following the
format444https://github.com/YoYo000/MVSNet/blob/master/mvsnet/colmap2mvsnet.py
defined in the open-source code of MVSNet.

Tanks&Temples (TT) [15] is a large-scale MVS dataset collected in outdoor
scenes with realistic conditions. It contains 7 scenes in the training set, 8
scenes in the Intermediate evaluation set, and 6 scenes in the Advanced
evaluation set. Following the preprocessed evaluation set provided by MVSNet,
we use both the Intermediate and Advanced sets for evaluation. However, no
processed training set is available in the open-source codes of MVS methods.
Hence, we further use COLMAP to sparsely reconstruct scenes on the training
set of TT, and separate the neighboring views based on the similarity of
matching points in different views. More importantly, the depth annotations
are not provided in the training set of TT, hence we render the densely
reconstructed point clouds of all scenes via COLMAP to obtain the depth map on
each view. Our processed training set of TT is used as a candidate for source
domains, and the evaluation set provided by MVSNet is used as a candidate for
target domain.

PASMVS (PA) [14] is a perfectly accurate synthetic MVS
dataset555https://github.com/andrebroekman/PASMVS, consisting of 400 scenes
with ground truth depth maps and camera parameters. Each scene is composed of
a unique combination of different camera focal lengths, geometrical models,
environmental textures, and materials. The material properties are primarily
specular, which simulate different light conditions with different diffuse
materials. Here we utilize the test set of PA as an extra evaluation dataset
while selecting target domains.

###  V-B MMD Difference among Datasets

In Fig. 1, the MMD distances among different source domains and target domains
are visualized to reveal the difference among various MVS datasets. To
calculate the MMD distance, we use a pre-trained VGG network to compute the
embedding feature of all images in the domain. Each image in the MVS
training/testing set is converted into a feature vector with 4096 channels.
All these feature vectors are concatenated together to construct a feature
matrix describing the corresponding domain. Then, we can calculate the MMD
distance with these feature matrices to assess the distribution gap among
different MVS datasets.

From the presented results in Fig. 1, several interesting phenomenons can be
found:

  * •

Compared with the training set of DT, the training set of BL has a much
smaller distribution gap towards the evaluation set of TT (BL $\rightarrow$
TT: 62.2; DT $\rightarrow$ TT: 108.1). This explains the fact that the
performance of MVS models trained on BL is better than the models trained on
DT, which is also revealed by previous works [10, 5]. Furthermore, it means
that the impressive generalization results of many recent methods on TT
benchmark may not be totally convincing, because the performance improvement
may be dependent on the benefit of the large-scale dataset with a small
distribution gap like BL, even the joint training of BL and DT. Consequently,
we investigate the domain generalization MVS problem in this paper and
restrict the setting of generalization with only one single domain to diminish
the aforementioned confounding factor.

  * •

The additional evaluation set of PA has much higher distribution gap compared
with other target domains. Note that PA simulates various conditions of object
materials and light conditions, which naturally possesses larger difference
compared with other target domains. Since DT models 7 different light
conditions in each scene, the distribution gap of DT $\rightarrow$ PA is much
smaller than the ones of other source domains. Consequently, the illuminance
is also an important factor that differs different MVS dataset between DT/PA
and BL/GS/TT.

Figure 4: Some examples of the processed ground truths of BL/GS/PA.

###  V-C 3D Ground Truth and Evaluation Benchmark

Although DT and TT contain their own 3D reconstruction evaluation benchmarks,
the remaining datasets BL, GS, and PA do not have similar benchmarks for 3D
reconstructed results. Hence, we also need to design customized quantitative
benchmarks to measure the performance of 3D reconstruction. To assess the
reconstruction quality quantitatively, collecting the 3D ground truth is
indispensable for the evaluation of MVS 3D reconstruction. Hence, we need to
prepare the 3D ground truth of each scene in the target domains and propose an
equitable evaluation metric to measure the reconstruction results.

####  V-C1 DT Evaluation

For DT, the ground truth and official evaluation benchmark are provided in the
official website666http://roboimagedata.compute.dtu.dk/?page_id=36. During the
evaluation, the metric of _Accuracy_ , _Completeness_ , and _Overall_ are used
to measure the reconstruction results. _Accuracy_ is used to measure the
average distance from the reconstructed point clouds to the ground truth point
clouds. _Completeness_ is used to measure the average distance from the ground
truth point clouds to the reconstructed point clouds. _Overall_ is the average
of _Accuracy_ and _Completeness_ , which is used as an overall assessment of
the reconstruction quality.

####  V-C2 TT Evaluation

For TT, the ground truth is not available, and the official evaluation
benchmark requires submission of reconstructed point clouds to their official
website777https://www.tanksandtemples.org/. The test set is separated into 2
parts: the _Intermediate_ set and the _Advanced_ set. _F-score_ on each scene
is used to measure the reconstruction quality.

####  V-C3 BL/GS/PA Evaluation

For BL/GS/PA, we have to prepare the ground truth and build the evaluation
benchmark by ourselves, since no existing evaluation benchmark is available in
the open-source community.

To get the 3D ground truth, we need to process BL/GS/PA differently:

  * •

For BL, the provided ground truth mesh is comprised of various splits. The
mesh parts are first merged to construct the complete mesh as ground truth.
Then, we utilize Monte-Carlo sampling to uniformly sample point cloud from the
ground truth mesh. The sampled point cloud in each scene is used as 3D ground
truth in the following evaluation stage. Some examples of the sampled ground
truth point cloud are shown in Fig. 4(a).

  * •

For GS, the ideally accurate depth maps and camera parameters are fused with
Gipuma [38] to construct the ground truth point cloud. It is noted that the
extrinsic matrix in GTASFM is the inverse form of the ones defined in Gipuma.
Some examples of the ground truth are visualized in Fig. 4(b).

  * •

For PA, the provided ground truth is 3d mesh template without texture. We
utilize Monte-Carlo sampling to uniformly sample the ground truth point cloud
from the original mesh. Some examples of the ground truth points are shown in
Fig. 4(c).

To construct the evaluation benchmark, we follow the evaluation protocol
defined by TT [15] and [58], which uses _Precision_ , _Recall_ , and _F-score_
to measure the reconstruction quality.

Denote that the ground truth point cloud is $P_{G}\in\mathbb{R}^{M\times 3}$,
and the reconstructed point cloud is $P_{R}\in\mathbb{R}^{N\times 3}$.
Following the definition of Chamfer Distance (CD), we can have:

$\displaystyle D_{CD}=\frac{1}{M}\sum_{p_{m}\in P_{G}}\min_{p_{n}\in
P_{R}}\|p_{m}-p_{n}\|_{2}^{2}$ (21) $\displaystyle+\frac{1}{N}\sum_{p_{n}\in
P_{R}}\min_{p_{m}\in P_{G}}\|p_{m}-p_{n}\|_{2}^{2}$

where $p_{m}$ is any point of ground truth point cloud $P_{G}$, and $p_{n}$ is
any point of reconstructed point cloud $P_{R}$. We can divide Eq. 21 into 2
terms:

$D_{G\rightarrow R}=\frac{1}{M}\sum_{p_{m}\in P_{G}}\min_{p_{n}\in
P_{R}}\|p_{m}-p_{n}\|_{2}^{2}$ (22) $D_{R\rightarrow
G}=\frac{1}{N}\sum_{p_{n}\in P_{R}}\min_{p_{m}\in
P_{G}}\|p_{m}-p_{n}\|_{2}^{2}$ (23)

where Eq. 22 represents the average distance of all points in the ground truth
point cloud $P_{G}$ towards the reconstructed point cloud $P_{R}$, which can
be viewed as a measure of completeness. And Eq. 23 represents the average
distance of all points in the reconstructed point cloud $P_{R}$ towards the
ground truth point cloud $P_{G}$, which can be treated as a measure of
accuracy.

The calculated distances can further be aggregated to define the recall and
precision given a threshold $d$:

$\text{Reca}(d)=\frac{100}{M}\sum_{p_{m}\in P_{G}}[\min_{p_{n}\in
P_{R}}\|p_{m}-p_{n}\|_{2}^{2}<d]$ (24)
$\text{Prec}(d)=\frac{100}{N}\sum_{p_{n}\in P_{R}}[\min_{p_{m}\in
P_{G}}\|p_{m}-p_{n}\|_{2}^{2}<d]$ (25)

where $[]$ is the Iverson bracket. The definition of _Recall_ and _Precision_
are respectively provided in Eq. 24 and Eq. 25.

The definition of _F-score_ can be calculated from _Recall_ and _Precision_ :

$\text{F-score}(d)=\frac{2\text{Prec}(d)\text{Reca}(d)}{\text{Prec}(d)+\text{Reca}(d)}$
(26)

Note that _F-score_ , _Precision_ , and _Recall_ are defined to lie in the
range of $[0-100]$. The higher these scores are, the better the performance
is.

###  V-D Network Details

We implement our method in PyTorch and train the models with 4 NVIDIA A-100
GPUs. The default setting is the same as the backbone network. For the hyper-
parameter settings, $\lambda=1.0$, $K=8$, $\epsilon=0.02$. The number of
adopted layers is set to 3. We train the network for 24 epochs on DT, BL, GS,
and 32 epochs on TT. Since the training set of TT is smaller than the others,
we maintain a fixed range of iterations by increasing the epochs a little. The
number of clusters on depth map is set to 8 in default. If not mentioned, the
adopted setting in training is the same as the one of the backbone MVS
network.

TABLE I: Quantitative comparisons with the-state-of-art MVS methods utilizing
DT as source domain. Note that all model is purely trained by DTU training
set, and tested on other datasets without any finetuning. Best results are in
bold, second best are underlined.

Methods | DT $\rightarrow$ DT | DT $\rightarrow$ BL | DT $\rightarrow$ GS | DT $\rightarrow$ PA
---|---|---|---|---
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
MVSNet [1] | 0.396 | 0.527 | 0.462 | 29.86 | 25.68 | 22.42 | 16.17 | 18.94 | 16.74 | 20.36 | 24.21 | 21.44
CasMVSNet [3] | 0.325 | 0.385 | 0.355 | 48.21 | 39.91 | 39.06 | 42.99 | 36.93 | 38.74 | 42.61 | 41.76 | 40.78
PatchMatchNet [59] | 0.427 | 0.277 | 0.352 | 41.15 | 49.34 | 41.69 | 49.47 | 38.25 | 42.66 | 30.12 | 40.42 | 33.95
IterMVS [60] | 0.373 | 0.354 | 0.363 | 47.21 | 43.93 | 41.79 | 47.36 | 36.91 | 40.92 | 45.00 | 44.84 | 44.37
MVSTER [47] | 0.350 | 0.276 | 0.313 | 41.29 | 40.60 | 36.26 | 24.86 | 18.64 | 20.75 | 33.50 | 52.28 | 40.45
CDS-MVSNet [61] | 0.352 | 0.280 | 0.316 | 43.74 | 33.97 | 33.89 | 22.45 | 22.71 | 21.66 | 29.09 | 20.92 | 23.72
UniMVSNet [5] | 0.352 | 0.278 | 0.315 | 49.09 | 35.06 | 37.93 | 54.30 | 30.66 | 38.47 | 64.46 | 22.07 | 32.24
RobustMVS-Cas | 0.328 | 0.365 | 0.346 | 52.07 | 42.91 | 42.72 | 45.45 | 44.08 | 43.85 | 56.74 | 49.99 | 52.03
RobustMVS-Iter | 0.315 | 0.406 | 0.361 | 49.17 | 43.97 | 43.13 | 51.32 | 39.11 | 43.83 | 49.93 | 44.87 | 45.89

Figure 5: Qualitative comparisons with state-of-the-art MVS methods utilizing
DT as source domain. TABLE II: Quantitative comparison with domain
generalization methods utilizing DT/BL/GS/TT as source domain. Best results
are in bold.

Methods | DT $\rightarrow$ DT | DT $\rightarrow$ BL | DT $\rightarrow$ GS | DT $\rightarrow$ PA
---|---|---|---|---
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
Baseline [3] | 0.325 | 0.385 | 0.355 | 48.21 | 39.91 | 39.06 | 42.99 | 36.93 | 38.74 | 42.61 | 41.76 | 40.78
FCStereo [27] | 0.3332 | 0.3673 | 0.3503 | 49.80 | 42.35 | 40.76 | 43.09 | 38.99 | 40.42 | 52.88 | 46.73 | 48.44
IBN [37] | 0.3284 | 0.3823 | 0.3554 | 46.22 | 42.35 | 39.52 | 46.20 | 41.34 | 42.73 | 55.32 | 49.26 | 50.80
EFDM [62] | 0.3218 | 0.3746 | 0.3482 | 45.78 | 41.67 | 39.35 | 46.02 | 40.44 | 42.19 | 56.16 | 47.98 | 50.71
RobustMVS-Cas | 0.3209 | 0.3634 | 0.3421 | 52.07 | 42.91 | 42.72 | 45.45 | 44.08 | 43.85 | 56.74 | 49.99 | 52.03
Methods | BL $\rightarrow$ DT | BL $\rightarrow$ BL | BL $\rightarrow$ GS | BL $\rightarrow$ PA
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
Baseline [3] | 0.4343 | 0.6874 | 0.5608 | 44.91 | 37.60 | 36.44 | 52.67 | 55.69 | 53.42 | 35.44 | 30.22 | 31.06
FCStereo [27] | 0.3534 | 0.3833 | 0.3684 | 44.03 | 43.87 | 38.65 | 53.44 | 55.32 | 53.69 | 37.29 | 31.86 | 32.51
IBN [37] | 0.3490 | 0.3880 | 0.3685 | 44.66 | 46.73 | 40.54 | 53.67 | 54.95 | 53.66 | 33.80 | 26.36 | 28.03
EFDM [62] | 0.3603 | 0.3844 | 0.3723 | 44.40 | 46.09 | 39.89 | 55.38 | 56.42 | 55.19 | 34.56 | 30.28 | 30.68
RobustMVS-Cas | 0.3484 | 0.3820 | 0.3652 | 47.06 | 46.41 | 41.82 | 56.14 | 56.33 | 55.50 | 40.03 | 36.101 | 36.09
Methods | GS $\rightarrow$ DT | GS $\rightarrow$ BL | GS $\rightarrow$ GS | GS $\rightarrow$ PA
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
Baseline [3] | 0.5083 | 0.9305 | 0.7194 | 43.20 | 33.51 | 33.15 | 52.59 | 55.27 | 53.23 | 38.63 | 25.48 | 28.81
FCStereo [27] | 0.4411 | 0.7490 | 0.5951 | 41.19 | 36.62 | 34.03 | 52.44 | 56.17 | 53.64 | 37.90 | 28.75 | 30.90
IBN [37] | 0.4531 | 0.7195 | 0.5863 | 44.87 | 37.42 | 36.49 | 50.92 | 56.72 | 52.85 | 45.83 | 32.82 | 36.37
EFDM [62] | 0.4084 | 0.5728 | 0.4906 | 44.12 | 38.33 | 36.80 | 51.00 | 56.14 | 52.76 | 34.66 | 27.01 | 28.47
RobustMVS-Cas | 0.3767 | 0.5390 | 0.4579 | 47.78 | 43.46 | 41.53 | 56.38 | 58.72 | 56.92 | 48.00 | 39.59 | 41.27
Methods | TT $\rightarrow$ DT | TT $\rightarrow$ BL | TT $\rightarrow$ GS | TT $\rightarrow$ PA
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
Baseline [3] | 0.4573 | 0.6505 | 0.5539 | 42.80 | 29.70 | 30.83 | 43.74 | 45.41 | 43.67 | 36.59 | 15.62 | 20.67
FCStereo [27] | 0.3721 | 0.5678 | 0.4700 | 44.52 | 30.02 | 31.17 | 43.53 | 45.80 | 43.83 | 38.27 | 17.64 | 22.79
IBN [37] | 0.4598 | 0.5917 | 0.5257 | 43.91 | 31.91 | 32.99 | 43.92 | 46.01 | 44.23 | 35.57 | 21.51 | 25.46
EFDM [62] | 0.3934 | 0.5635 | 0.4784 | 40.63 | 30.27 | 30.79 | 40.37 | 44.64 | 41.54 | 32.71 | 14.76 | 19.32
RobustMVS-Cas | 0.3687 | 0.5320 | 0.4504 | 47.01 | 32.45 | 34.33 | 46.42 | 46.95 | 47.13 | 43.12 | 25.20 | 30.21

TABLE III: Generalization results to TT evaluation benchmark.

Methods |  DT $\rightarrow$ TT |  BL $\rightarrow$ TT
---|---|---
Intermediate↑ | Advanced↑ | Intermediate↑ | Advanced↑
COLMAP [63] | 42.14 | 27.24 | 42.14 | 27.24
MVSNet [1] | 43.38 | - | - | -
R-MVSNet [6] | 48.4 | 24.91 | - | -
CIDER [64] | 46.76 | 23.12 | - | -
PatchMatchNet [59] | 53.15 | 32.31 | - | -
PatchMatchRL [65] | - | - | 51.81 | 31.78
VisMVSNet [53] | - | - | 60.03 | 33.78
IterMVS [60] | 56.22 | 33.24 | 56.94 | 34.17
CasMVSNet [3] | 56.84 | 31.12 | 57.22 | 32.63
TransMVSNet [10] | 55.0 | 30.2 | 60.3 | 35.3
RobustMVS-Cas | 57.53 | 32.68 | 58.49 | 33.61
RobustMVS-Trans | 57.43 | 31.17 | 60.80 | 36.66

###  V-E Comparison with MVS methods

In Table I, we compare the quantitative performance on domain generalization
with other state-of-the-art MVS methods. Only the training set of DT is
selected as source domain to train the model, and we evaluate the performance
on other source domains. We directly utilize the open-sourced pre-trained
model of CasMVSNet [3], PatchMatchNet [59], IterMVS [60], MVSTER [47], CDS-
MVSNet [61], and UniMVSNet [5], to test on target domains without finetuning.
These pre-trained models are further used for evaulation on unseen datasets on
DT $\rightarrow$ BL, DT $\rightarrow$ GS, and DT $\rightarrow$ PA. We further
provide experimental results on the same dataset (DT $\rightarrow$ DT) to
evaluate the performance. In the table, RobustMVS-Cas means that the CasMVSNet
is selected as the backbone of our RobustMVS framework. In analogy, RobustMVS-
Iter means that the IterMVS is selected as the backbone network. The metric of
F-score reflects the overall performance of 3D reconstruction. The best
results are in bold, and the second best is marked with underline.

An interesting phenomenon can be found that these state-of-the-art models in
comparison may fail to generalize to other domains (BL, GS, PA) well even with
impressive performance on _DT_. The reason is that their parameters are highly
tuned to fit well-used datasets _DT_ or TT with human efforts, resulting in
overfitting effect to some extent when a novel unseen domain in involved.
Smaller models like CasMVSNet, PatchMatchNet, and IterMVS suffer less from the
overfitting issue. Comparing with state-of-the-art MVS methods, our RobustMVS-
Cas and RobustMVS-Iter achieves better performance on all 3 cross-domain
tracks. As the results show, our proposed method can effectively reduce the
performance degradation in unseen novel domains. Comparing with the baseline
method with same backbone (CasMVSNet / IterMVS), our RobustMVS-Cas and
RobustMVS-Iter can improve the generalization performance effectively on all 3
cross-domain tracks. It shows that our proposed RobustMVS can improve the
generalization performance when combined with different MVS backbones. For
qualitative results, we provide the visualization comparisons in Fig. 5. Some
examples under DT $\rightarrow$ BL and DT $\rightarrow$ GS are selected for
visualization. The reconstructed point clouds of RobustMVS-Cas is compared
with CDS-MVSNet and MVSTER in the figure. From the figure, we can find that
our method can achieve more accurate and complete reconstruction than these
state-of-the-art MVS methods. Furthermore, the proposed method can achieve
slightly better reconstruction performance on RobustMVS-Cas and RobustMVS-Iter
compared with the baseline CasMVSNet and IterMVS. It shows that our RobustMVS
can preserve great performance in the same domain (DT $\rightarrow$ DT)
meantime increasing the generalization aility towards other domains (DT
$\rightarrow$ BL/GS/PA).

###  V-F Comparison with DG methods

In Table II, we provide the quantitative comparison between our method and
other domain generalization methods. For comparison, we extend the
implementation of Stereo-Selective Whitening (SSW) in FCStereo [27] to the
setting of MVS. FCStereo is a pioneering work in domain generalized stereo
with 2 camera settings, which provides a robust baseline for comparison in the
table. IBN [37] and EFDM [58] are state-of-the-art domain generalization
modules that can be directly plugged into the backbone network. Since our
proposed RobustMVS only insert several instance normalization layers into the
backbone network, it can provide a direct and fair comparison between our
proposed method and these plug-in domain generalized modules. The same
backbone network of CasMVSNet [3] is used. We report the quantitative results
of cross-domain generalization MVS on 16 different tracks in the table. The
source domain and target domain is respectively selected from: DT, BL, GS, and
PA. From the direct comparison between FCStereo and our method, our method can
significantly improve the reconstruction performance that provides a better
multi-view feature-whitening based solution for domain generalization in MVS.
Moreover, our method can also outperform other domain generalized baselines
like IBN and EFDM on the metric of _Overall_ in DT and _F-score_ in BL, GS,
and PA. The qualitative results presents the comparison on some reconstructed
samples in Fig. 6. Some examples of GS $\rightarrow$ DT and GS $\rightarrow$
PA are shown in the figure. From the figure, it can be found that our method
can reconstruct better 3D point cloud compared with other method, and suffer
less from the distribution gap when generalizing from synthetic domain GS to
realistic domains.

TABLE IV: Improvement of generalization results to TT evaluation benchmark
with the same backbone, CasMVSNet [3].

Methods |  DT $\rightarrow$ TT |  BL $\rightarrow$ TT
---|---|---
Intermediate↑ | Advanced↑ | Intermediate ↑ | Advanced↑
Baseline [3] | 56.84 | 31.12 | 57.22 | 32.63
RobustMVS-Cas ($\Delta$↑) | +0.69 | +1.56 | +1.27 | +0.98
Methods |  GS $\rightarrow$ TT |  TT $\rightarrow$ TT
Intermediate↑ | Advanced↑ | Intermediate ↑ | Advanced↑
Baseline [3] | 44.65 | 22.61 | 47.07 | 26.55
RobustMVS-Cas ($\Delta$↑) | +8.91 | +7.1 | +2.85 | +0.95

TABLE V: Ablation results of different components in RobustMVS.

Components |  DT $\rightarrow$ BL |  DT $\rightarrow$ GS |  DT $\rightarrow$ PA
---|---|---|---
Baseline | DCW | IN | DA | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
$\checkmark$ |  |  |  | 48.21 | 39.91 | 39.06 | 42.99 | 36.93 | 38.74 | 42.61 | 41.76 | 40.78
$\checkmark$ | $\checkmark$ |  |  | 48.20 | 41.44 | 39.70 | 44.33 | 39.28 | 40.65 | 51.31 | 46.49 | 47.51
$\checkmark$ | $\checkmark$ | $\checkmark$ |  | 51.80 | 42.07 | 41.94 | 47.10 | 37.69 | 41.03 | 47.04 | 57.95 | 51.55
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 52.07 | 42.91 | 42.72 | 45.45 | 44.08 | 43.85 | 56.74 | 49.99 | 52.03

TABLE VI: Ablation experiments of the number of depth clusters $K$. GS
(synthetic data) is used as source domain.

Depth clusters | GS $\rightarrow$ DT | GS $\rightarrow$ BL | GS $\rightarrow$ GS | GS $\rightarrow$ PA
---|---|---|---|---
Acc.↓ | Comp.↓ | Overall↓ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑ | Prec.↑ | Reca.↑ | F-score↑
K=0 | 0.4411 | 0.7490 | 0.5951 | 41.19 | 36.62 | 34.03 | 52.44 | 56.17 | 53.64 | 37.90 | 28.75 | 30.90
K=4 | 0.4148 | 0.4148 | 0.5122 | 44.92 | 40.00 | 38.53 | 54.02 | 57.91 | 55.27 | 39.72 | 33.59 | 34.63
K=6 | 0.3886 | 0.6315 | 0.5101 | 46.51 | 42.28 | 40.40 | 55.51 | 58.53 | 56.41 | 42.27 | 36.11 | 37.44
K=8 | 0.3767 | 0.5390 | 0.4579 | 47.78 | 43.46 | 41.53 | 56.38 | 58.72 | 56.92 | 48.00 | 39.59 | 41.27
K=10 | 0.4087 | 0.5993 | 0.5040 | 43.51 | 38.24 | 36.19 | 52.64 | 56.07 | 53.67 | 42.11 | 33.25 | 35.34
K=12 | 0.4141 | 0.6085 | 0.5113 | 41.00 | 38.24 | 35.62 | 52.45 | 57.08 | 54.03 | 33.45 | 30.64 | 30.88
K=14 | 0.4372 | 0.5953 | 0.5162 | 40.93 | 36.64 | 34.35 | 50.78 | 56.34 | 52.71 | 33.84 | 26.88 | 27.89

Figure 6: Qualitative comparison with domain generalization methods utilizing
GS as source domain.

###  V-G Generalization on Tanks&Temples

The generalization results on TT evaluation set are provided in Table III.
F-score is used to measure the performance of reconstruction results. The
intermediate and advanced sets of TT are both utilized for evaluation
following previous works [60, 10]. The compared methods include: COLMAP [63],
MVSNet [1], R-MVSNet [6], CIDER [64], PatchMatchNet [59], PatchMatchRL [65],
VisMVSNet [53], IterMVS [60], CasMVSNet [3], and TransMVSNet [10]. The open-
sourced pretrained model of the compared methods are utilized for evaluation.
RobustMVS-Cas and RobustMVS-Trans respectively adopts CasMVSNet and
TransMVSNet as the backbone network in our RobustMVS framework. The criteria
for choosing thse networks as baselines is to compare the single domain
generalization performance in a comparative setting. Since TransMVSNet adopts
multiple datasets (DT \+ BL) for training, we use their official
implementation to retrain the network on single domain respectively to ensure
a fair comparison in the single domain generalization benchmark. In the table,
the performance of DT $\rightarrow$ TT and BL $\rightarrow$ TT are reported
for comparison. As the results show, our RobustMVS-Trans outperforms the
baseline TransMVSNet on the intermediate and advanced set in both 2 settings.
Superior results can also be witnessed in the comparison between RobustMVS-Cas
and the baseline CasMVSNet. When compared with other methods, our RobustMVS
can still have competitive performance and even better results.

In Table IV, we further provide the improvement of generation results towards
TT dataset in evaluation. The same backbone of CasMVSNet is adopted here. The
evaluation includes 4 different tracks: DT $\rightarrow$ TT, BL $\rightarrow$
TT, GS $\rightarrow$ TT, and TT $\rightarrow$ TT. It can be explored that the
RobustMVS can improve the performance on all these benchmarks in both the
intermediate and advanced set. Furthermore, when the synthetic dataset GS is
used as source domain, significant improvement can be witnessed on
intermediate set (+8.91) and advanced set (+7.1).

Figure 7: Grad-cam [16] visualization example of RobustMVS under domain
generalization setting of GS $\rightarrow$ DT.

###  V-H Ablation Study

1) _Ablation Analysis of Different Components_ : To evaluate the effectiveness
of each component in the proposed RobustMVS framework, we conduct ablation
experiments on them and provide the quantitative results in Table V. In the
table we present the results on 3 different tracks: DT $\rightarrow$ Bl, DT
$\rightarrow$ GS, andDT $\rightarrow$ PA. The backbone network is CasMVSNet in
default. The ”baseline” represents that the original CasMVSNet network. ”DCW”
means the proposed DCW-loss is appended in the training phase. ”IN” and ”DA”
respectively represents the instance normalization module in the network and
data augmentation strategy during training. As the table shows, when DCW-loss
is appended to the MVS network training, the F-score is improved on all 3
benchmarks. Since the Instance Normalization is usually paired with the
whitenning loss, better performance can be achieved when IN is also used. The
involvement of data-augmentation strategy enables the feature whitening loss
to reduce the domain bias under more various scenarios compared with the naive
setting. As the table shows, each component is effective in improving the
generalization performance.

2) _Ablation Analysis of Different Clusters_ : To explore the effectiveness of
the depth clusters $K$ in DCW loss, we also conduct ablation experiments for
different settings: ${K_{\text{clu}}}=0/4/6/8/10/12/14$ on the benchmark of GS
$\rightarrow$ DT, GS $\rightarrow$ BL, GS $\rightarrow$ GS, and GS
$\rightarrow$ PA. When ${K_{\text{clu}}}=0$, it represents the basic feature
whitening loss used in FCStereo [27]. When ${K_{\text{clu}}}$ is larger than
0, it represents the depth cluster used in our DCW-loss. As shown in Table VI,
the performance improves when ${K_{\text{clu}}}$ is smaller than 8 but
degrades when ${K_{\text{clu}}}$ becomes larger than 8. The reason is that
${K_{\text{clu}}}$ is small, increasing the number of clusters can regularize
the cross-view correspondence in a more detailed local region. However, when
${K_{\text{clu}}}$ becomes too large, the pixels located in the local
corresponding regions contains less variance of style information, which tends
to have smaller difference among each other. Thus the regularization on these
small regions can not effectively excavate the style information when
calculating feature whitening loss, and degrade the reconstruction
performance.

There are a few suggestions if you want to adjust the depth cluster number
($K_{\text{clu}}$) when the image resolution (H, W) changes: a) If the image
resolution is similar in scale to the default setting in our synthetic domain
($640\times 480$ in GS), you can use the default setting of
($K_{\text{clu}}$), i.e. DT ($640\times 512$). b) If the difference between
the image resolutions of the target dataset and the default synthetic dataset
GS is too large, we suggest downsampling the image resolution to a comparative
scale of the default one. For example, the image resolution of TT can be
resized to $640\times 480$ from its original one for training. c) The depth
cluster number choice may also be affected by the variety of depth values in a
scene (synthetic or real). This is because the balanced distance-based metric
in K-Means may be affected by the sparsity of point clouds. For example, for
some sharp shapes vertical to the image plane, the sampled points from depth
might be pretty sparse. In point clouds, K-Means clustering might ignore these
sparse points and focus on the dense points with flat shapes horizontal to the
image plane. This problem might occur when the image resolution is relatively
small. We can mitigate it by not shrinking the image resolution. d) We suggest
not setting $K_{\text{clu}}$ to be larger than 8 as training might become
unstable and return NaN values caused by the strong whitening constraints.
Finally, due to the implementation issue in parallelism, we utilize an even
number of depth clusters for parallelizing the computation.

3) _Grad-cam visualization of MVS_ : To explain the effectiveness of the
proposed RobustMVS framework, we provide the Grad-cam [16] visualization
results in Fig. 7. We compare the baseline model of CasMVSNet and RobustMVS-
Cas for visualization. In default the source domain is synthetic dataset GS
and the target domain is DT. The error between the estimated depth map of the
MVS network and the ground truth depth is back-propagated to the convolutional
layers, we utilize Grad-cam [16] to visualize the gradient on feature maps. In
the first column, the original images of the reference view and source view
are shown. The second and third column respectively show the activation map of
baseline model and our proposed method. The blue circle in the figure points
out the ignored activation areas in the baseline model. For example, in the
left example, the light exposure results in textureless regions on the red
ball, which has never been seen before in the synthetic dataset GS.
Consequently, the baseline model can not handle this case and these areas are
not activated in blue circles. However, our RobustMVS can provide more
complete activation map compared with baseline model on the whole activation
map. From these examples, we can find that our RobustMVS provide a robust
regularization to reduce the domain bias and maintain a well-distributed
feature activation map across domains.

TABLE VII: Comparison of computation efficiency. The inference time is
evaluated on a single A100 GPU.

Methods | Params | GFLOPS | Inference Time (s)
---|---|---|---
CasMVSNet | 934.304K | 268.540G | 0.1904
RobustMVS-Cas | 934.192K | 268.401G | 0.1977

4) _Computation efficiency_ : We have compared the baseline of CasMVSNet with
our RobustMVS-Cas, and the results are presented in Table VII. Both networks
use the same backbone, but our RobustMVS-Cas replaces some BN layers with IN
layers. Our RobustMVS-Cas has slightly fewer parameters and lower GFLOPS than
the baseline backbone. However, the inference time of RobustMVS-Cas might be
slightly slower than the baseline backbone network. This is because the
original backbone uses batch normalization layers that can store the running
mean and variance values, reducing computation during inference. In contrast,
our RobustMVS-Cas requires extra online computation during inference due to
the replacement of some batch normalization layers with instance normalization
layers. Despite the slightly slower inference time than the baseline, our
RobustMVS-Cas can significantly improve the generalization performance.

##  VI Conclusion

This paper first introduces the domain generalization problem in the field of
MVS. As the visualization of the activation map on MVS networks across
different domains shows, the bottleneck lies in the robustness towards cross-
view matching features across domain-specific conditions. Consequently, we
propose a novel domain generalized MVS framework, RobustMVS. It requires
little modifications to the backbone MVS network and applies a Depth-
Clustering-guided Whitening (DCW) loss to diminish the domain-specific styles
on spatially neighboring regions. We extend the MVS evaluation benchmarks to 5
datasets for domain generalization and compare the proposed method with
existing MVS methods and domain generalization methods. The experimental
results reveal the robust performance of the proposed method in the domain
generalization benchmark of MVS. The final visualization of activation maps in
our proposed method gives clues to indicate whether it can reduce the
influence of domain bias. For future work, we will explore the further
extension of Transformer-based MVS networks in the domain generalized MVS
tasks and try to conduct domain generalization from large-scale synthetic MVS
datasets towards realistic MVS datasets.

## References

  * [1] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for unstructured multi-view stereo,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 767–783.
  * [2] R. T. Collins, “A space-sweep approach to true multi-image matching,” in _Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. Ieee, 1996, pp. 358–363.
  * [3] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan, “Cascade cost volume for high-resolution multi-view stereo and stereo matching,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2495–2504.
  * [4] S. Cheng, Z. Xu, S. Zhu, Z. Li, L. E. Li, R. Ramamoorthi, and H. Su, “Deep stereo using adaptive thin volume representation with uncertainty awareness,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2524–2534.
  * [5] R. Peng, R. Wang, Z. Wang, Y. Lai, and R. Wang, “Rethinking depth estimation for multi-view stereo: A unified representation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8645–8654.
  * [6] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan, “Recurrent mvsnet for high-resolution multi-view stereo depth inference,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5525–5534.
  * [7] T. Khot, S. Agrawal, S. Tulsiani, C. Mertz, S. Lucey, and M. Hebert, “Learning unsupervised multi-view stereopsis via robust photometric consistency,” _arXiv preprint arXiv:1905.02706_ , 2019.
  * [8] B. Huang, H. Yi, C. Huang, Y. He, J. Liu, and X. Liu, “M3vsnet: Unsupervised multi-metric multi-view stereo network,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 3163–3167.
  * [9] H. Xu, Z. Zhou, Y. Qiao, W. Kang, and Q. Wu, “Self-supervised multi-view stereo via effective co-segmentation and data-augmentation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 4, 2021, pp. 3030–3038.
  * [10] Y. Ding, W. Yuan, Q. Zhu, H. Zhang, X. Liu, Y. Wang, and X. Liu, “Transmvsnet: Global context-aware multi-view stereo network with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8585–8594.
  * [11] R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs, “Large scale multi-view stereopsis evaluation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 406–413.
  * [12] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan, “Blendedmvs: A large-scale dataset for generalized multi-view stereo networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1790–1799.
  * [13] K. Wang and S. Shen, “Flow-motion and depth network for monocular stereo and beyond,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3307–3314, 2020.
  * [14] A. Broekman and P. J. Gräbe, “Pasmvs: a perfectly accurate, synthetic, path-traced dataset featuring specular material properties for multi-view stereopsis training and reconstruction applications,” _Data in brief_ , vol. 32, p. 106219, 2020.
  * [15] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and temples: Benchmarking large-scale scene reconstruction,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, pp. 1–13, 2017.
  * [16] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [17] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of representations for domain adaptation,” _Advances in neural information processing systems_ , vol. 19, 2006.
  * [18] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [19] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in _International conference on machine learning_. Pmlr, 2018, pp. 1989–1998.
  * [20] X. Ye, J. Zhang, Y. Yuan, R. Xu, Z. Wang, and H. Li, “Underwater depth estimation via stereo adaptation networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [21] K. Muandet, D. Balduzzi, and B. Schölkopf, “Domain generalization via invariant feature representation,” in _International Conference on Machine Learning_. PMLR, 2013, pp. 10–18.
  * [22] H. Li, S. J. Pan, S. Wang, and A. C. Kot, “Domain generalization with adversarial feature learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5400–5409.
  * [23] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao, “Deep domain generalization via conditional invariant adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 624–639.
  * [24] C. Cai, M. Poggi, S. Mattoccia, and P. Mordohai, “Matching-space stereo networks for cross-domain generalization,” in _2020 International Conference on 3D Vision (3DV)_. IEEE, 2020, pp. 364–373.
  * [25] F. Zhang, X. Qi, R. Yang, V. Prisacariu, B. Wah, and P. Torr, “Domain-invariant stereo matching networks,” in _European Conference on Computer Vision_. Springer, 2020, pp. 420–439.
  * [26] Z. Shen, Y. Dai, and Z. Rao, “Cfnet: Cascade and fused cost volume for robust stereo matching,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 13 906–13 915.
  * [27] J. Zhang, X. Wang, X. Bai, C. Wang, L. Huang, Y. Chen, L. Gu, J. Zhou, T. Harada, and E. R. Hancock, “Revisiting domain generalized stereo matching networks from a feature consistency perspective,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 001–13 011.
  * [28] H. Dai, X. Zhang, Y. Zhao, H. Sun, and N. Zheng, “Adaptive disparity candidates prediction network for efficient real-time stereo matching,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 5, pp. 3099–3110, 2022.
  * [29] Z. Lu, J. Wang, Z. Li, S. Chen, and F. Wu, “A resource-efficient pipelined architecture for real-time semi-global stereo matching,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 2, pp. 660–673, 2022.
  * [30] B. Pan, L. Zhang, and H. Wang, “Multi-stage feature pyramid stereo network-based disparity estimation approach for two to three-dimensional video conversion,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 5, pp. 1862–1875, 2021.
  * [31] L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [32] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2414–2423.
  * [33] X. Pan, X. Zhan, J. Shi, X. Tang, and P. Luo, “Switchable whitening for deep representation learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1863–1871.
  * [34] P. Luo, “Learning deep architectures via generalized whitened neural networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2238–2246.
  * [35] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _European conference on computer vision_. Springer, 2016, pp. 443–450.
  * [36] S. Choi, S. Jung, H. Yun, J. T. Kim, S. Kim, and J. Choo, “Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 580–11 590.
  * [37] X. Pan, P. Luo, J. Shi, and X. Tang, “Two at once: Enhancing learning and generalization capacities via ibn-net,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 464–479.
  * [38] S. Galliani, K. Lasinger, and K. Schindler, “Massively parallel multiview stereopsis by surface normal diffusion,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 873–881.
  * [39] N. D. Campbell, G. Vogiatzis, C. Hernández, and R. Cipolla, “Using multiple hypotheses to improve depth-maps for multi-view stereo,” in _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10_. Springer, 2008, pp. 766–779.
  * [40] Y. Furukawa and J. Ponce, “Accurate, dense, and robust multiview stereopsis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 8, pp. 1362–1376, 2010.
  * [41] J. Yan, Z. Wei, H. Yi, M. Ding, R. Zhang, Y. Chen, G. Wang, and Y.-W. Tai, “Dense hybrid recurrent multi-view stereo net with dynamic consistency checking,” in _European conference on computer vision_. Springer, 2020, pp. 674–689.
  * [42] Z. Wei, Q. Zhu, C. Min, Y. Chen, and G. Wang, “Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6187–6196.
  * [43] J. Yang, W. Mao, J. M. Alvarez, and M. Liu, “Cost volume pyramid based depth inference for multi-view stereo,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4877–4886.
  * [44] W. Su, Q. Xu, and W. Tao, “Uncertainty guided multi-view stereo network for depth estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 11, pp. 7796–7808, 2022.
  * [45] R. Zhao, X. Han, X. Guo, L. Kuang, X. Yang, and F. Sun, “Exploring the point feature relation on point cloud for multi-view stereo,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 11, pp. 6747–6763, 2023.
  * [46] H. Zhang, X. Ye, S. Chen, Z. Wang, H. Li, and W. Ouyang, “The farther the better: Balanced stereo matching via depth-based sampling and adaptive feature refinement,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4613–4625, 2021.
  * [47] X. Wang, Z. Zhu, G. Huang, F. Qin, Y. Ye, Y. He, X. Chi, and X. Wang, “Mvster: Epipolar transformer for efficient multi-view stereo,” in _European Conference on Computer Vision_. Springer, 2022, pp. 573–591.
  * [48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [49] H. Xu, Z. Zhou, Y. Wang, W. Kang, B. Sun, H. Li, and Y. Qiao, “Digging into uncertainty in self-supervised multi-view stereo,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6078–6087.
  * [50] A. Mallick, J. Stückler, and H. P. Lensch, “Learning to adapt multi-view stereo by self-supervision.” in _BMVC_ , vol. 1, 2020, p. 2.
  * [51] S. Chen, Z. Pu, X. Fan, and B. Zou, “Fixing defect of photometric loss for self-supervised monocular depth estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1328–1338, 2021.
  * [52] Y. Wei, H. Guo, J. Lu, and J. Zhou, “Iterative feature matching for self-supervised indoor depth estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 6, pp. 3839–3852, 2021.
  * [53] J. Zhang, Y. Yao, S. Li, Z. Luo, and T. Fang, “Visibility-aware multi-view stereo network,” _arXiv preprint arXiv:2008.07928_ , 2020.
  * [54] W. Cho, S. Choi, D. K. Park, I. Shin, and J. Choo, “Image-to-image translation via group-wise deep whitening-and-coloring transformation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 639–10 647.
  * [55] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Universal style transfer via feature transforms,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [56] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016.
  * [57] J. A. Hartigan and M. A. Wong, “Algorithm as 136: A k-means clustering algorithm,” _Journal of the royal statistical society. series c (applied statistics)_ , vol. 28, no. 1, pp. 100–108, 1979.
  * [58] H. Xu, Z. Zhou, W. Cheng, B. Sun, H. Li, and W. Kang, “Semi-supervised deep multi-view stereo,” _arXiv preprint arXiv:2207.11699_ , 2022.
  * [59] F. Wang, S. Galliani, C. Vogel, P. Speciale, and M. Pollefeys, “Patchmatchnet: Learned multi-view patchmatch stereo,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 194–14 203.
  * [60] F. Wang, S. Galliani, C. Vogel, and M. Pollefeys, “Itermvs: Iterative probability estimation for efficient multi-view stereo,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8606–8615.
  * [61] K. T. Giang, S. Song, and S. Jo, “Curvature-guided dynamic scale networks for multi-view stereo,” _arXiv preprint arXiv:2112.05999_ , 2021.
  * [62] Y. Zhang, M. Li, R. Li, K. Jia, and L. Zhang, “Exact feature distribution matching for arbitrary style transfer and domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8035–8045.
  * [63] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4104–4113.
  * [64] Q. Xu and W. Tao, “Learning inverse depth regression for multi-view stereo with correlation cost volume,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 508–12 515.
  * [65] J. Y. Lee, J. DeGol, C. Zou, and D. Hoiem, “Patchmatch-rl: Deep mvs with pixelwise depth, normal, and visibility,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6158–6167.

|  Hongbin Xu received B.E. degree from Nanchang University in 2018 and M.S.
degree from South China University of Technology in 2021. He is currently
pursuing the Ph.D. degree with the School of Automation Science and
Engineering, South China University of Technology, Guangzhou, China. His
research interests include 3D vision, 3D reconstruction, and computer vision.
---|---
|  Weitao Chen Weitao Chen received the M.S degress from Xiamen University,
Xiamen, China, in 2016. He is working as a staff/senior algorithm engineer at
Tongyi Laboratory, Alibaba Group. His research interests include 3D vision,
AIGC, AI4Science and remote sensing.
---|---
|  Baigui Sun Baigui Sun received the M.Eng. degree from Zhejiang University,
Hangzhou, China, in 2014. He joined Alibaba Group, Hangzhou, China, as a
computer vision algorithm researcher, in 2014. His research interests include
face detection/recognition, deep metric learning, domain adaption, zero/few-
shot learning, semi-supervised learning, MVS and computer vision applications.
He has published over 26+ articles in international journals and conferences.
---|---
|  Xuansong Xie Xuansong Xie received the PH.D. degree from Jilin University,
Jilin, China. He is a senior algorithm expert at Alibaba, serving as the
technical director of the Open Vision Intelligence Department at the Tongyi
Laboratory. He is also a core member of the Alibaba Group’s visual technology
team and responsible for the Alibaba Cloud Vision Intelligence Open Platform.
His research interests include visual generation, editing, perception and
understanding.
---|---
|  Wenxiong Kang (Member, IEEE) received the M.S. degree from Northwestern
Polytechnical University, Xi’an, China, in 2003, and the Ph.D. degree from the
South China University of Technology, Guangzhou, China, in 2009. He is
currently a Professor with the School of Automation Science and Engineering,
South China University of Technology. His research interests include
biometrics identification, image processing, pattern recognition, and computer
vision.
---|---