11institutetext:  Jagiellonian University, Faculty of Mathematics and Computer
Science 22institutetext: Jagiellonian University, Doctoral School of Exact and
Natural Sciences 33institutetext: IDEAS NCBR

#  A deep cut into Split Federated Self-supervised Learning

Marcin Przewięźlikowski 112233 0000-0003-4772-3268 Marcin Osial 112233
0000-0002-7414-3608 Bartosz Zieliński 1133 0000-0002-3063-3621 Marek Śmieja 11
0000-0003-2027-4132

###### Abstract

Collaborative self-supervised learning has recently become feasible in highly
distributed environments by dividing the network layers between client devices
and a central server. However, state-of-the-art methods, such as MocoSFL, are
optimized for network division at the initial layers, which decreases the
protection of the client data and increases communication overhead. In this
paper, we demonstrate that splitting depth is crucial for maintaining privacy
and communication efficiency in distributed training. We also show that
MocoSFL suffers from a catastrophic quality deterioration for the minimal
communication overhead. As a remedy, we introduce Momentum-Aligned contrastive
Split Federated Learning (MonAcoSFL), which aligns online and momentum client
models during training procedure. Consequently, we achieve state-of-the-art
accuracy while significantly reducing the communication overhead, making
MonAcoSFL more practical in real-world scenarios111Our codebase is available
at https://github.com/gmum/MonAcoSFL.

###### Keywords:

Federated learning Self-supervised learning Contrastive learning

##  1 Introduction

Collaborative learning techniques allow multiple participating parties to
jointly train models without compromising the confidentiality of their private
data, making them increasingly popular [28, 31, 33, 38, 24]. Among these
techniques, Federated Learning (FL) [28] stands out as the most prevalent
framework. In FL, the learning task is solved by a federation of participating
devices (which we refer to as clients) coordinated by a central server. Each
client has a local training dataset (unseen by the server) and computes an
update to the current global model maintained by the server (only this update
is communicated). Moreover, a part of the model can be trained on the server
side, resulting in a setup called Split Federated Learning (SFL) [31].

Federated learning has exhibited considerable success in supervised learning
tasks [28, 10, 32]. However, the assumption of full labeling may sometimes be
impractical due to the challenges and expertise required for accurate labeling
[27]. That is why more practical methods focus on unlabeled data [37, 38],
combining FL with classic self-supervised learning (SSL) [16]. A notable
example is MocoSFL, which is based on Split Federated Learning (SFL) and
Momentum Contrast (MoCo) [17, 24].

MocoSFL can achieve good accuracy with relatively low memory requirements and
can support a large number of clients. However, it was optimized for network
division at the initial layers (one or three layers on client devices), which
has certain disadvantages. First, it decreases the protection of the client
data (Figure 2). Secondly, it increases communication overhead (information
transferred between clients and server), as shown in Figure 1.

In this paper, we delve into the relationship between communication overhead
and the splitting point. We identify the optimal splitting point and highlight
the poor performance of MocoSFL when aligned with it. As a remedy, we
introducing Momentum-Aligned Contrastive SFL (MonAcoSFL). In contrast to
MocoSFL, which synchronizes only the online client models during the training,
MonAcoSFL also synchronizes their momentum models. This change is crucial
because it prevents the divergence of online and momentum models and reduces
confusion during training (see Figure 5).

Figure 1: Communication overhead and accuracy for MocoSFL and MonAcoSFL
depending on the splitting depth. In this example, the minimal communication
overhead is obtained for dividing the model into 11 and 7 layers on the client
and server sides, respectively. For such an optimal case, the accuracy of
MocoSFL drops significantly, in contrast to the accuracy of MonAcoSFL. Notice
that computational overhead consists of forward and backward propagation
(marked as blue) and parameter synchronization (marked as orange).

Through extensive experiments, we empirically confirm that the synchronizing
scheme of MocoSFL indeed creates misalignment between its online and momentum
models, which is not the case in MonAcoSFL. Crucially, we show that when
training mobile-friendly architectures split at deeper, more communication-
efficient points, MonAcoSFL improves over the performance of MocoSFL [24] by a
significant margin, confirming its practicality.

Our contributions can be summarized as follows:

  * •

We analyze the practical trade-off in communication efficiency of split
federated learning and show that MocoSFL performs poorly for the most optimal
splits.

  * •

We re-think model synchronization in MocoSFL and introduce MonAcoSFL, which
reduces the divergence between online and momentum clients at each
synchronization step.

  * •

We present an extensive experimental study and confirm that MonAcoSFL
maintains high performance even when applied to deeper splits of the backbone
model, increasing its practical applicability to real-world scenarios.

##  2 Related work

Federated learning (FL) [28, 36, 26, 25] is a method of training models across
multiple decentralized entities or devices while keeping the data localized,
without the need to send it to a centralized location [28]. Models are
individually trained and then aggregated by a global server, ensuring data
privacy by sharing only model parameters among clients. The seminal FedAVG
algorithm [28], which conducts a straightforward weighted averaging of
clients’ model weights, remains a solid foundational framework for numerous
approaches [37, 38, 23]. One of the most important variants of FL is Split
Federated Learning (SFL), which involves storing a part of the trained model
on a centralized server, reducing the computational overhead of clients at the
cost of additional communication overhead due to sending latent vectors from
clients to the server [31]. In addition, SFL creates a trade-off between the
clients’ energy consumption and privacy [22].

Self-supervised learning (SSL), a pivotal development in unsupervised
representation learning, shines particularly in its joint-embedding form
within the realm of computer vision [8, 9, 17, 4]. This approach leverages
contrastive learning [3, 29] to modulate similarity and disparity among
augmented views. To prevent trivial solutions, such methods often rely on
substantial memory banks, like MoCo [17, 12], or necessitate large batch sizes
as observed with SimCLR [11], to amass a significant number of meaningful
negative examples. On the other hand, non-contrastive methods such as BYOL
[16] do not require negative examples in their objectives, yet still require
large batch sizes during training [16, 13, 9]. A common architectural
component of several popular SSL approaches is the maintaining of "momentum"
models to achieve asymmetry of representations [17, 12, 14, 9, 2].

Early attempts at deploying the SSL methods in Federated setting have grappled
with the complexity of data diversity [5] and addressed key privacy
considerations [34]. Methods like FedU [37] utilize the BYOL framework for
federated SSL, while FedEMA [38] refines this by adaptively updating client
networks with insights from the global model’s trends. However, the above
methods cannot be scaled beyond cross-silo settings, involving up to 100
client devices. Li et.al. demonstrate the key ingredients of scaling SSL to
highly distributed cross-client environments (as many as 1000 clients) in the
form of MocoSFL – a combination of MoCo and the Split Federated Learning
paradigm [24].

##  3 Preliminaries

In this section, we first describe the problem of federated learning in a
self-supervised regime and its challenges. Then, we recall the MocoSFL [24], a
recently published method, which we use as a basis for our MonAcoSFL.

###  3.1 Federated self-supervised learning

#### Problem statement.

In this paper, we tackle the problem of _federated self-supervised learning_.
It considers multiple participating devices (called clients) and a centralized
server that collaboratively trains a single model based on data gathered by
the former. However, the devices are prohibited from sending their raw data to
the server e.g. for privacy reasons. Instead, the training is conducted in a
_federated learning_ manner, where each client trains a copy of the model on
its local data and periodically synchronizes it with other clients. Another
assumption is that data gathered by the devices do not contain labels. Thus,
the model must be trained by optimizing a _self-supervised_ objective. After
the training, the resulting model can be finetuned to the target tasks using
labeled data and distributed to all participating devices.

#### Split federated learning

(SFL) [28] is a practical approach to federated learning, which splits the
model into client and server parts. For a formal definition, let us define $N$
as the number of participating clients and $f_{\phi_{i}}$ as a copy of model
$f_{\phi}$ stored by the $i$-th client parametrized by $\phi_{i}$.
Furthermore, let $f_{\phi}$ be composed of two parts $f^{s}_{\phi^{s}}\circ
f^{c}_{\phi^{c}}$, where $\phi=\phi^{s}\cup\phi^{c}$, $f^{c}_{\phi^{c}}$ is
distributed among client devices, and $f^{s}_{\phi^{s}}$ is stored on a
centralized server. There are multiple copies of $f^{c}_{\phi^{c}}$ but only
one version of parameters222Note that FL is a special case of SFL where
$f=f^{c}$, whereas a situation where $f=f^{s}$ is equivalent to regular
training on a single device. $\phi^{s}$.

Client models start with the same initial parameters
($\phi^{c}_{1}=\phi^{c}_{2}=\dots=\phi^{c}_{N}$) and are trained in two
phases:

  1. 1.

Optimization of $\phi^{c}_{1},...,\phi^{c}_{N},\phi^{s}$ w.r.t. to training
objective $\mathcal{L}$.

  2. 2.

Synchronization of $\phi^{c}_{1},...,\phi^{c}_{N}$ by overwriting each
$\phi^{c}_{i}$ with $\hat{\phi^{c}}=\sum_{i=0}^{N}\dfrac{\phi^{c}_{i}}{N}$.

Each client processes only its local data during the optimization step, which
leads to increasingly diverging $\phi^{c}_{1},...,\phi^{c}_{N}$. That is why
the synchronization step is required.

In standard Federated learning, the communication is limited to transferring
parameters for model synchronization. In SFL, we must also transfer from
clients the activations of $f^{c}$ (for forward propagation by the server) and
their gradients w.r.t. to $\mathcal{L}$ from the $f^{s}$ residing on the
server (for backpropagation in local models).

#### Self-supervised learning

(SSL) is a paradigm of learning representations without data labels [1, 4].
Currently, the prevalent approaches to SSL are the joint-embedding
architectures [17, 12, 11, 16, 9, 2], where model $f$ learns by optimizing
_contrastive objectives_. Formally, let
$\mathbf{x}^{\prime},\mathbf{x}^{\prime\prime}$ be two augmentations of a
sample $\mathbf{x}\sim X$. Contrastive objectives enforce the similarity of
$f(\mathbf{x}^{\prime})$ and $f(\mathbf{x}^{\prime\prime})$ while avoiding
trivial solutions, i.e., producing identical embeddings of unrelated data
samples. For this purpose, most joint-embedding methods [11, 9] use objective
functions requiring large batch sizes.

Due to the large-data requirements and the computational overhead associated
with the contrastive objectives, deploying SSL methods in highly distributed
Federated environments is challenging in practice [37, 38, 24].

###  3.2 Momentum contrastive split federated learning

Momentum Contrastive Split Federated Learning (MocoSFL) [24] addresses the
practical challenges of deploying SSL methods in distributed environments by
combining SFL [31] with Momentum Contrastive Learning (MoCo) [17] method.

Similarly to MoCo, the contrastive objective of MocoSFL is calculated with
InfoNCE [29] based on the _memory of embeddings_ $M$:

$\mathcal{L}_{InfoNCE}(\mathbf{z}^{\prime},\mathbf{z}^{\prime\prime},M)=-log\frac{exp(\mathbf{z}^{\prime}\cdot\mathbf{z}^{\prime\prime}/\tau)}{exp(\mathbf{z}^{\prime}\cdot\mathbf{z}^{\prime\prime}/\tau)+\sum_{\mathbf{z}_{M}\in
M}exp(\mathbf{z}^{\prime}\cdot\mathbf{z}_{M}/\tau)}$ (1)

where $\mathbf{z}^{\prime}=f_{\phi}(\mathbf{x}^{\prime})$,
$\mathbf{z}^{\prime\prime}=f_{EMA(\phi)}(\mathbf{x}^{\prime\prime})$,
$\mathbf{x}^{\prime}$ and $\mathbf{x}^{\prime\prime}$ are two augmentations of
a single data sample, and $\mathbf{z}_{M}$ denotes the embeddings stored in
the memory $M$ (after each training step, $M$ is updated with
$\mathbf{z}^{\prime\prime}$ in a first-in-first-out manner). Moreover, $\phi$
are parameters of the _online_ model, and $EMA(\phi)$, being the exponential
moving average of $\phi$, are parameters of the _momentum_ model. Because
MocoSFL operates in the SFL setup, each client contains its version of
parameters $\phi^{c}_{i}$ and $EMA(\phi^{c}_{i})$, and there is one version of
parameters $\phi^{s}$ and $EMA(\phi^{s})$ on the server. Similarly, the memory
$M$ is also maintained by the server.

To our best knowledge, MocoSFL is the only SSL method operating in a
challenging cross-client federated learning with over 100 clients, each
contributing as few as 250 data samples from different distributions [24]. It
is possible by using a large memory of negative examples from all clients,
which minimizes the detrimental effect of potentially small batch sizes of
individual clients [7, 11] and reducing the chance of overfitting to any
individual client distribution [35]. Moreover, relegating most model layers
and the contrastive objective to the server reduces the computational burden
on the clients [31].

##  4 MonAcoSFL: Momentum-Aligned Contrastive Split Federated Learning

In this section, we first describe the limitations of MocoSFL and show that
its performance deteriorates using higher split layers (see Section 4.1).
Then, we analyze the source of this deterioration and introduce MonAcoSFL as a
proposed solution (see Section 4.2).

###  4.1 Limitations of MocoSFL

The main limitation of MocoSFL is a drop in performance when splitting the
network at the higher layers. In consequence, it can be effectively used only
with a few layers on the client side. It leads to two crucial concerns. First,
sending representations (activations) from low layers makes the model more
sensitive to data leakage and attacks such as the Model Inversion Attack (MIA)
[15]. Second, low-layer representations are much larger than those returned by
higher layers, which increases communication overhead. Below, we describe
those problems in detail.

#### Privacy concerns

caused by sending representations from low layers is illustrated in Figure 2.
One can observe that representations of low ResNet18 [18] layers highly
resemble the respective input data, in contrast to the activations from the
higher layers. In fact, in principle, models that learn perceptive features
(such as MoCo) do not retain reconstructive features in their high
representations [4]. Thus, in SFL, increasing the number of layers on the
client side reduces the privacy risks associated with broadcasting network
representations.

Figure 2: Activations obtained from successive layers of untrained ResNet-18
for a sample ImageNet image. One can observe that representations of low
layers highly resemble the respective input data, increasing the privacy risks
associated with broadcasting network activations.

#### Communication overhead.

Network architectures used in mobile devices, such as ResNet [18] or MobileNet
[30], progressively downsample the spacial dimensions of representations
obtained from successive layers while simultaneously increasing their channel
dimensions. As a result, the overall representation size _decreases_ with
network depth, and the communication in the SFL optimization phase requires
less bandwidth. However, more layers on client devices require exchanging more
parameters during the SFL synchronization phase, which translates to a larger
bandwidth. Therefore, a trade-off exists between those two communication
overheads, as illustrated in Figure 3 with an optimal split in the higher
layers.

Figure 3: Communication overhead of a single client device for one training
epoch of ResNet-18 [18] and MobileNet-V2 [30] for different splitting depths.
The 11-th and 7-th layers are the most communication-efficient for ResNet-18
and MobileNetV2, respectively. Note that the training epoch corresponds to
$250$ images of resolution $224\times 224$ are processed, and $10$
synchronizations of parameters. Moreover, the blue bars correspond to
communication in the optimization phase, and the orange bars correspond to
parameter synchronization.

#### Deteriorated performance for higher split layer.

Based on the above considerations, it is evident that choosing a larger
splitting depth can be attractive due to privacy and efficiency reasons. Thus,
a natural question arises – how does splitting depth affect the performance of
MocoSFL? To answer this, we benchmark its performance on the CIFAR-10 dataset
[20] for 5, 20, and 200 clients, following the experimental setup of [24] (see
Section 5.1) and report the results in Figure 4 333Note that Li et. al. [24]
evaluate MocoSFL only with split layer set to 1 or 3.. Surprisingly, for
higher split layers, we observe a catastrophic deterioration in the accuracy
of the trained models.

To summarize, pretraining mobile-friendly architectures with MocoSFL with a
splitting depth larger than $7$ is unreliable, and using a lower splitting
depth poses concerns to privacy and communication overhead.

Figure 4: Accuracy of MocoSFL drops significantly with increased splitting
depth regardless of the number of clients. Here, presented for CIFAR-10 [20].

###  4.2 MonAcoSFL

Before providing our solution to the limitations of MocoSFL, we will first
outline the main reason behind them. For this purpose, let us analyze the
MocoSFL training procedure in detail. As we described in Section 3.1, client
models start with the same initial parameters
$\phi^{c}_{1}=\phi^{c}_{2}=\dots=\phi^{c}_{N}$. However, during training, they
move away from one another due to different local datasets, and they must be
periodically synchronized to stabilize the training. Like in regular SFL
schemes, MocoSFL synchronizes _only the online client models_. However, such
rapid modification of online parameters breaks the underlying assumption of
MoCo that respective online and momentum models encode similar representations
at all times, which is essential for minimizing the contrastive objective
[17]. We illustrate this phenomenon on the left side of Figure 5. This problem
grows with the increased splitting depth because more parameters become
misaligned, which explains the decrease in performance.

Figure 5: Visualization of parameters changing in MocoSFL (left) and MonAcoSFL
(right) for two clients. The solid and dashed lines represent the progression
of online and momentum parameters, respectively. The dotted lines symbolize
the synchronization of parameters. The difference between MocoSFL and
MonAcoSFL lies in the synchronization procedure that, in the case of
MonAcoSFL, ensures that both online and the momentum models remain aligned,
preserving their ability to optimize the contrastive objective.

To overcome this limitation, we propose Momentum-Aligned Contrastive SFL
(MonAcoSFL) presented on the right side of Figure 5. It ensures the alignment
of respective online and momentum client models by synchronizing the
respective client momentum models whenever the online models are synchronized,
i.e. by overwriting momentum model of each client with:

$\widehat{EMA(\phi^{c})}=\dfrac{\sum_{i=0}^{N}EMA(\phi^{c}_{i})}{N}$

Note that since $EMA(\phi^{c}_{1}),\dots,EMA(\phi^{c}_{N})$ are the EMAs of
the individual $\phi^{c}_{1},\dots,\phi^{c}_{N}$, their average always
corresponds to the EMA of the average value of the online parameters
($\hat{\phi^{c}}$), i.e: $\widehat{EMA(\phi^{c})}=EMA(\hat{\phi^{c}}).$

##  5 Experimental investigation

In this section, we experimentally evaluate MonAcoSFL and compare it with
MocoSFL, a recent state-of-the-art method in self-supervised federated
learning. We evaluate both models using classification accuracy on downstream
tasks and verifying the privacy of clients data. Finally, we experimentally
analyze the alignment of online and momentum models in MoCo framework. For a
fair comparison, we closely follow the experimental setup of Li et.al. [24].

###  5.1 Experimental setup

#### Hardware

We simulate the distributed environment of MocoSFL and MonAcoSFL on a single
machine that hosts the client and server models and runs all of them on a
single NVidia A100 GPU. We conduct experiments on mobile-friendly ResNet18
[18] and MobileNetV2 [30] backbones.

#### Data

We conduct our experiments on CIFAR-10 and CIFAR-100 [20] datasets. We divide
the samples equally between the clients to simulate the situation where each
client has access to only a small part of the data. We assume a challenging
setting where the data is not Independent and Identically Distributed (non-
IID) between the client devices. Namely, we assign for each client images from
randomly chosen 2 (for CIFAR-10) or 20 (for CIFAR-100) classes.

#### Number of clients

We compare MocoSFL and MonAcoSFL in both cross-silo (5 or 20 clients) and
cross-client (200 clients) settings. To our knowledge, MocoSFL is the only
previous self-supervised federated method able to scale to the cross-client
setting. We synchronize the models on client devices 10 times per training
epoch, which amounts to synchronizing the model every 1000, 250, and 25
encountered images for the 5, 20, and 200-client settings, respectively. We
adjust the client-side batch size and client sampling ratio accordingly in
order to keep the server-side batch size at around 100. For example, in the
5-client setting, the batch size is 20 and all clients are used at each
training epoch, whereas in the 200-client setting, we use a random choice of
100 clients in each epoch and therefore each client has a batch size of 1.

#### SSL model

We use the MoCo-v2 [12], where the backbone is succeeded by a 2-layer MLP
projector network with a hidden size of 1024 [11, 6], discarded after SSL
pretraining. The server maintains a memory $M$ of 6000 negative embeddings,
implemented as a First-In-First-Out (FIFO) queue. After each training step,
the queue is updated with the momentum model embeddings of the most recent
mini-batch of samples. We train MocoSFL for 200 epochs, with the SGD
optimizer, with an initial learning rate of 0.06, 0.9 momentum, and 0.0005
weight decay. The learning rate is scheduled throughout the training according
to the cosine scheduler.

#### Evaluation

After each epoch, we perform the k-NN validation of the model using 20% of the
validation dataset. K-NN validation is the standard method of evaluating self-
supervised representations during their training [21, 24]. After training, we
use the model which achieved the best k-NN performance. We measure the final
performance of the model using the widely used linear evaluation protocol [11,
16, 38, 21, 24]. Namely, we attach random liner layer to the frozen pretrained
backbone and train only this layer on the labeled dataset for 100 epochs, with
a batch size of 128 and the Adam optimizer [19] with initial learning rate of
0.001 and cosine learning rate scheduler.

###  5.2 Accuracy performance

Figure 6 presents the accuracy of MonAcoSFL and MocoSFL in the cross-silo (5
or 20 clients) and cross-client (200 clients) settings on the ResNet
architecture [20]. Although both models decrease the accuracy with increasing
splitting depth, this decrease is much smaller in MonAcoSFL than in MocoSFL.
In the most communication-efficient splitting point (layer 11-13), the
difference between MonAcoSFL and MoCoSFL exceeds 30 percentage points of
accuracy. An even stronger advantage of MonAcoSFL is confirmed on the
MobileNet backbone, see Figure 7. In the case of 20 clients, MonAcoSFL
improves the accuracy of MoCoSFL by more than 40 percentage points starting
from 3th to 15th cut layer on the CIFAR-10 dataset.

Figure 6: Linear evaluation of MocoSFL and MonAcoSFL on ResNet18 architecture.
MonAcoSFL maintains the accuracy with increasing cut-layers, whereas the
performance of MocoSFL rapidly deteriorates. Figure 7: Linear evaluation
accuracy on MobileNetV2 backbone trained with MocoSFL and MonAcoSFL. There is
significant discrepancy between MonAcoSFL and MoCoSFL at every splitting
depth.

###  5.3 Privacy Evaluation

We next compare the privacy-preserving capabilities of MonAcoSFL and MoCoSFL
with a Model Inversion Attack (MIA) [15]. We assume that the attacker has
access to 1% of the training data and trains a decoder for reconstructing
images from client model embeddings. We train the decoder with the MSE loss
and Adam optimization with a learning rate of 0.001 over 50 epochs and a batch
size of 32. We perform an attack on ResNet-18 networks pretrained on CIFAR-100
dataset with 20 clients and varying cut-layers. The attack is conducted
without pre-training or employing additional privacy-enhancing techniques like
TAResSFL [24].

We compare the original and reconstructed images in terms of MSE in Figure
8(b), where a higher MSE indicates better privacy protection. We find that for
cut-layers 1-13 the attacks on MocoSFL and MonAcoSFL yield similar
reconstruction errors, with the errors of attacks on MonAcoSFL being larger by
a small margin. The MSE remains relatively stable for layers 1-7 and increases
for layers 9-17, indicating greater privacy of embeddings of these cut-layers.
In layers 15-17, MocoSFL achieves larger attack MSE than MonAcoSFL. We
attribute this to the fact that the representation of MocoSFL trained with
these cut-layers is in general of much worse quality (which results in lower
accuracy, see Figure 6) and, as a side-effect, encodes less information about
the data. Concluding, using deeper cut-layers (9-13) is not only more
efficient from the computational point of view but also increases the
protection of client data.

(a) Images reconstructed by the attacker at different layers following MIA on
both MocoSFL and MonAcoSFL.

(b) MSE of reconstructing the original images by the attacker. Higher MSE
values for deeper cut-layers indicates a better resistance to attack (better
privacy).

Figure 8: MIA attack on ResNet-18 models trained with MocoSFL and MonAcoSFL.
Deeper splits provide better protection of client data. Both methods achieve
similar levels of privacy protection.

###  5.4 Analysis of models alignment

To empirically verify that MonAcoSFL indeed preserves the alignment of online
and momentum model parameters, we measure it throughout the training. We
define the average misalignment of online and momentum parameters as their
average absolute difference i.e.:

$\sum_{i=0}^{N}\dfrac{|\phi^{c}_{i}-EMA(\phi^{c}_{i})|}{N\cdot
dim(\phi^{c})},$ (2)

where $N$ denotes the number of clients and $dim(\phi^{c})$ is the dimension
of client model parameters.

We plot in Figure 9 the misalignment values for 1500 initial training steps
(out of approximately 50000) of ResNet18 trained on CIFAR-100 by 20 clients,
MocoSFL and MonAcoSFL, with split at the 11th layer. Throughout the first 125
steps, both methods display similar misalignments. However, during parameter
synchronizations, the difference between the online and momentum models in
MocoSFL rapidly increases by an order of magnitude. On the other hand, in
MonAcoSFL, the misalignment of momentum and online models does not change
significantly throughout the training.

Figure 9: A comparison of average misalignment of online and momentum client
models in the initial steps of training for MonAcoSFL and MocoSFL (lower is
better). We mark with blue lines the steps at which the parameter
synchronization is performed (every 125 steps). In MocoSFL the misalignment
grows rapidly during parameter synchronizations, whereas in MonAcoSFL it
remains an order of magnitude smaller, resulting in a more stable training.

##  6 Conclusion

In this paper, we conducted an in-depth analysis of MocoSFL – the state-of-
the-art method for Federated Self-Supervised Learning [24]. We found that the
quality of this method deteriorates in privacy-preserving and communication-
efficient settings and identified that the reason for this is its
synchronization scheme, which leaves the participating online and momentum
models misaligned. We addressed this problem by introducing Momentum-Aligned
contrastive Split Federated Learning (MonAcoSFL) and showed that it
significantly improves the performance of MoCoSFL, especially under
communication-efficient conditions.

From a theoretical perspective, our solution is centered on the correct
synchronization of online and momentum models of self-supervised learners
deployed in distributed environments. Given that the online/momentum model
pairs are a staple of several modern SSL approaches [16, 9, 2], our findings
can be applicable in federated learning of various SSL methods other than MoCo
[17]. From the practical perspective, MonAcoSFL is the first Federated self-
supervised method to achieve good results using deeper cut layers, providing
efficient communication between clients and server and better protection of
client data.

## Ethical implications

In this work, we focus on improving the existing Federated self-supervised
approaches in regimes where data privacy and communication efficiency are of
concern. Our findings can lead to reducing the risks of data leakages when
deploying Federated self-supervised learning algorithms in real-world
scenarios.

## Acknowledgements

This research has been supported by the flagship project entitled "Artificial
Intelligence Computing Center Core Facility" from the Priority Research Area
DigiWorld under the Strategic Programme Excellence Initiative at Jagiellonian
University, and by the Horizon Europe Programme (HORIZON-CL4-2022-HUMAN-02)
under the project "ELIAS: European Lighthouse of AI for Sustainability", GA
no. 101120237. The research of Marcin Przewięźlikowski was supported by the
National Science Centre (Poland), grant no. 2023/49/N/ST6/03268. The research
of Marcin Osial was supported by National Science Centre (Poland) grant number
2023/50/E/ST6/00469. The research of Marek Śmieja was supported by the
National Science Centre (Poland), grant no. 2022/45/B/ST6/01117. The research
of Bartosz Zieliński was supported by National Science Centre (Poland) grant
number 2022/47/B/ST6/03397. Some experiments were performed on servers
purchased with funds from a grant from the Priority Research Area (Artificial
Intelligence Computing Center Core Facility) under the Strategic Programme
Excellence Initiative at Jagiellonian University. We gratefully acknowledge
Polish high-performance computing infrastructure PLGrid (HPC Center: ACK
Cyfronet AGH) for providing computer facilities and support within
computational grant no. PLG/2023/016303.

## References

  * [1] Albelwi, S.: Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging. Entropy 24(4) (2022). https://doi.org/10.3390/e24040551, https://www.mdpi.com/1099-4300/24/4/551
  * [2] Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15619–15629 (June 2023)
  * [3] Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual information across views. Advances in neural information processing systems 32 (2019)
  * [4] Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A.S., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A.G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., Goldblum, M.: A cookbook of self-supervised learning. ArXiv abs/2304.12210 (2023), https://api.semanticscholar.org/CorpusID:258298825
  * [5] van Berlo, B., Saeed, A., Ozcelebi, T.: Towards federated unsupervised representation learning. In: Proceedings of the third ACM international workshop on edge systems, analytics and networking. pp. 31–36 (2020)
  * [6] Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., Vincent, P.: Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. Transactions on Machine Learning Research (2023), https://openreview.net/forum?id=ZgXfXSz51n
  * [7] Bulat, A., Sánchez-Lozano, E., Tzimiropoulos, G.: Improving memory banks for unsupervised learning with large mini-batch, consistency and hard negative mining. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1695–1699 (2021). https://doi.org/10.1109/ICASSP39728.2021.9414389
  * [8] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 9912–9924. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf
  * [9] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * [10] Chen, C., Zhou, J., Zheng, L., Wu, H., Lyu, L., Wu, J., Wu, B., Liu, Z., Wang, L., Zheng, X.: Vertically federated graph neural network for privacy-preserving node classification. In: International Joint Conference on Artificial Intelligence (IJCAI). pp. 1959–1965 (2022)
  * [11] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR (13–18 Jul 2020), https://proceedings.mlr.press/v119/chen20j.html
  * [12] Chen, X., Fan, H., Girshick, R.B., He, K.: Improved baselines with momentum contrastive learning. CoRR abs/2003.04297 (2020), https://arxiv.org/abs/2003.04297
  * [13] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15750–15758 (June 2021)
  * [14] Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9640–9649 (October 2021)
  * [15] Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. p. 1322–1333. CCS ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2810103.2813677, https://doi.org/10.1145/2810103.2813677
  * [16] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 21271–21284. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
  * [17] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [18] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017)
  * [20] Krizhevsky, A.: Learning multiple layers of features from tiny images (2009)
  * [21] Lee, H., Lee, K., Lee, K., Lee, H., Shin, J.: Improving transferability of representations via augmentation-aware self-supervision. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 17710–17722. Curran Associates, Inc. (2021), https://proceedings.neurips.cc/paper_files/paper/2021/file/94130ea17023c4837f0dcdda95034b65-Paper.pdf
  * [22] Lee, J., Seif, M., Cho, J., Poor, H.V.: Exploring the privacy-energy consumption tradeoff for split federated learning (2024)
  * [23] Lee, R., Kim, M., Li, D., Qiu, X., Hospedales, T., Huszár, F., Lane, N.D.: Fedl2p: Federated learning to personalize. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=FM81CI68Iz
  * [24] Li, J., Lyu, L., Iso, D., Chakrabarti, C., Spranger, M.: MocoSFL: enabling cross-client collaborative self-supervised learning. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=2QGJXyMNoPz
  * [25] Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 (2021)
  * [26] Liu, Q., Chen, C., Qin, J., Dou, Q., Heng, P.A.: Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1013–1023 (2021)
  * [27] Makhija, D., Ho, N., Ghosh, J.: Federated self-supervised learning for heterogeneous clients (2023), https://openreview.net/forum?id=bNPth9YMqZ
  * [28] McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.y.: Communication-Efficient Learning of Deep Networks from Decentralized Data. In: Singh, A., Zhu, J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282. PMLR (20–22 Apr 2017), https://proceedings.mlr.press/v54/mcmahan17a.html
  * [29] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [30] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4510–4520. IEEE Computer Society, Los Alamitos, CA, USA (jun 2018). https://doi.org/10.1109/CVPR.2018.00474, https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00474
  * [31] Thapa, C., Mahawaga Arachchige, P.C., Camtepe, S., Sun, L.: Splitfed: When federated learning meets split learning. Proceedings of the AAAI Conference on Artificial Intelligence 36(8), 8485–8493 (Jun 2022). https://doi.org/10.1609/aaai.v36i8.20825, https://ojs.aaai.org/index.php/AAAI/article/view/20825
  * [32] Wu, C., Wu, F., Lyu, L., Qi, T., Huang, Y., Xie, X.: A federated graph neural network framework for privacy-preserving personalization. Nature Communications 13(1), 3091 (2022)
  * [33] Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., Gao, Y.: A survey on federated learning. Knowledge-Based Systems 216, 106775 (2021). https://doi.org/https://doi.org/10.1016/j.knosys.2021.106775, https://www.sciencedirect.com/science/article/pii/S0950705121000381
  * [34] Zhang, F., Kuang, K., Chen, L., You, Z., Shen, T., Xiao, J., Zhang, Y., Wu, C., Wu, F., Zhuang, Y., et al.: Federated unsupervised representation learning. Frontiers of Information Technology & Electronic Engineering 24(8), 1181–1193 (2023)
  * [35] Zhang, F., Kuang, K., You, Z., Shen, T., Xiao, J., Zhang, Y., Wu, C., Zhuang, Y., Li, X.: Federated unsupervised representation learning (2020)
  * [36] Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 (2018)
  * [37] Zhuang, W., Gan, X., Wen, Y., Zhang, S., Yi, S.: Collaborative unsupervised visual representation learning from decentralized data. In: IEEE/CVF International Conference on Computer Vision (CVPR). pp. 4912–4921 (2021)
  * [38] Zhuang, W., Wen, Y., Zhang, S.: Divergence-aware federated self-supervised learning. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=oVE1z8NlNe


# Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on
Disparity

Qingsong Yan ,1,2, Qiang Wang ,3, Kaiyong Zhao 4, Bo Li 2, Xiaowen Chu ,2,5,
Fei Deng †,1 Corresponding author

###### Abstract

Existing learning-based multi-view stereo (MVS) methods rely on the depth
range to build the 3D cost volume and may fail when the range is too large or
unreliable. To address this problem, we propose a disparity-based MVS method
based on the epipolar disparity flow (E-flow), called DispMVS, which infers
the depth information from the pixel movement between two views. The core of
DispMVS is to construct a 2D cost volume on the image plane along the epipolar
line between each pair (between the reference image and several source images)
for pixel matching and fuse uncountable depths triangulated from each pair by
multi-view geometry to ensure multi-view consistency. To be robust, DispMVS
starts from a randomly initialized depth map and iteratively refines the depth
map with the help of the coarse-to-fine strategy. Experiments on DTUMVS and
Tanks&Temple datasets show that DispMVS is not sensitive to the depth range
and achieves state-of-the-art results with lower GPU memory.

##  1 Introduction

Multi-view stereo matching (MVS) is a core technique in 3D reconstruction that
has been extensively studied (Goesele et al. 2007; Furukawa and Ponce 2009;
Galliani et al. 2015; Schönberger et al. 2016). Although traditional methods
try to introduce additional constraints (Xu and Tao 2019; Romanoni and
Matteucci 2019; Xu et al. 2020; Wang et al. 2020; Xu and Tao 2020b) to deal
with textureless regions or repeated textures, they still have difficulty in
guaranteeing the generation of high-quality point clouds in many cases.

Figure 1:  The influence of the depth range. This figure compares DispMVS with
two state-of-the-art methods ( GBiNet (Mi, Di, and Xu 2022) and IterMVS (Wang
et al. 2022) ) and shows that DispMVS can still generate high-quality point
clouds when changing the depth range by Eq. 23. Unlike other methods that use
the depth range to build a 3D cost volume and estimate the depth map, DispMVS
only builds a 2D cost volume along the epipolar line and uses the multi-view
geometry to estimate the depth map.

Recently, learning-based methods have brought a new light to MVS. MVSNet (Yao
et al. 2018) shows a fully differentiable pipeline, which firstly uses a
convolutional neural network (CNN) to extract features from input images,and
then splits the 3D space into several bins covering a certain depth range to
build a 3D cost volume by differentiable homograph, and finally relies on a 3D
CNN to regress the depth map. Although MVSNet achieves impressive results on
several public benchmarks (Aanæs et al. 2016; Knapitsch et al. 2017), it is
not efficient and requires a lot of GPU memory to infer. Therefore, the
coarse-to-fine strategy (Gu et al. 2020; Yang et al. 2020; Cheng et al. 2020;
Mi, Di, and Xu 2022) and the recurrent network (Yao et al. 2019; Wei et al.
2021; Wang et al. 2021, 2022) are used to upgrade the MVSNet. The coarse-to-
fine strategy uses the coarse stage to recover a coarse depth map and reduces
the number of bins needed at the fine stage to reduce the GPU memory.
Furthermore, the recurrent network, such as LSTM (Shi et al. 2015) and GRU
(Cho et al. 2014), can directly replace the 3D CNN to process the 3D cost
volume with a lower GPU memory footprint during the inference. Since these
strategies may sacrifice accuracy due to the reduced search range and the lack
of global context, many studies have improved the design of loss functions and
feature processing modules. In terms of loss functions, there is not only the
regression-based method to guide continuous depth (Yao et al. 2018; Gu et al.
2020) but also the classification-based method to guide discrete depth through
3D cost volume (Yao et al. 2019; Mi, Di, and Xu 2022), or a mixture of them to
achieve better depth prediction quality (Peng et al. 2022). As for the feature
processing module, the deformation convolution (Wei et al. 2021), the
attention mechanism (Luo et al. 2020), and the Transformer (Zhu et al. 2021)
are used to improve the quality of image features. In addition, the visibility
information (Xu and Tao 2020c; Zhang et al. 2020) and the epipolar (Ma et al.
2021; Yang et al. 2022) are also used to boost the performance.

A critical issue of those existing methods is the fixed depth range in
building the cost volume (Cheng et al. 2020; Mi, Di, and Xu 2022). Usually,
the depth range decides the 3D distribution of cost volume that the network
attempts to fit, and the size of cost volume is limited to the computational
and memory capability, which results in these methods can be easy to over-fit
the configured depth range. Fig. 1 shows that the quality of point cloud
construction by two state-of-the-art methods, GBiNet (Mi, Di, and Xu 2022) and
IterMVS (Wang et al. 2022), is dramatically degraded when the depth range is
enlarged. The reason is that these methods cannot capture enough matching
information with a fixed number of depth bins.

In this paper, we propose a new MVS pipeline, which allows the CNN to focus
only on the matching problem between two different views and relies on the
multi-view geometry to recover the depth by matching results. The
contributions of this paper are as follows.

  * •

Instead of constructing the 3D cost volume, this paper only constructs the 2D
cost volume to match pixels between each pair and generates the depth map by
triangulation. In other words, DispMVS exploits the multi-view geometry to
reduce the burden of networks, which does not rely on the computationally
expensive 3D cost volume to estimate the depth.

  * •

We redesign the flow to deal with the multi-view stereo matching without
applying stereo rectification for each pair. First, we propose the epipolar
disparity flow (E-flow) and reveal the relationship between the E-flow and the
depth. Then we extend E-flow from two-view to multi-view and iteratively
update the E-flow by a fused depth to maintain multi-view consistency.

  * •

DispMVS achieves the state-of-the-art result on the DTUMVS and Tanks&Temple
without the 3D cost volume, demonstrating the effectiveness of combining
multi-view geometry with a learning-based approach.

##  2 Related Work

With decades of development of MVS, many traditional and learning methods are
proposed. Traditional MVS cannot surpass the limitations of the artificially
designed matching pipeline and fail to reconstruct the non-Lambertian regions.
On the contrary, learning-based methods can automatically find the most
helpful information in a data-driven manner, and the benchmarking results of
(Aanæs et al. 2016; Knapitsch et al. 2017) show that the learning-based method
can easily outperform traditional methods. Generally, the learning-based
methods build a 3D cost volume, and we can categorize them into 3D
convolution-based and RNN-based methods according to how they handle the 3D
cost volume.

#### Traditional MVS

Traditional MVS has three major types: volumetric method, point cloud method,
and depth map method. The volumetric method (Seitz and Dyer 1999; Kutulakos
and Seitz 2000; Kostrikov, Horbert, and Leibe 2014) splits the space into
inside and outside and cuts out the surface. The point cloud method (Lhuillier
and Quan 2005; Goesele et al. 2007; Furukawa and Ponce 2009) reconstructs
dense point clouds from sparse point clouds. Although these methods can
reconstruct high-quality results, the volumetric and point cloud methods
require lots of GPU memory and are hard to parallelize. The depth map method
is the most popular, which separately reconstructs the depth map of each view
and fuses them to generate the point cloud (Merrell et al. 2007; Galliani et
al. 2015). Shen (Shen 2013) and Colmap (Zheng et al. 2014; Schönberger et al.
2016) extends the PatchMatch (Bleyer, Rhemann, and Rother 2011) to multi-view
and simultaneously estimates the normal and the depth. Meanwhile, Gipuma
(Galliani et al. 2015) and ACMM (Xu and Tao 2019) use a GPU to improve the
computational efficiency. The superpixel (Romanoni and Matteucci 2019), the
plane (Xu and Tao 2020b), and the mesh (Wang et al. 2020) are used to reduce
mismatching in non-Lambertian regions. Although these methods can achieve
stable results, they cannot surpass the limitation of hand-craft methods in a
challenging environment.

#### 3D Convolution Method

Various approaches have been proposed to address the shortcomings of MVSNet
(Yao et al. 2018).

The straightforward solution to reduce the GPU memory is building fewer 3D
cost volume. Fast-MVSNet (Yu and Gao 2020) only calculates the 3D cost volume
on a sparse depth map and propagates the sparse depth map into a dense depth
map. CasMVSNet (Gu et al. 2020) and CVP-MVSNet (Yang et al. 2020) uses a
coarse-to-fine strategy to deal with the 3D cost volume and reduce computation
cost on the high resolution. GBiNet (Mi, Di, and Xu 2022) treats the MVS as a
binary search problem and only builds the 3D cost volumes on the side with a
high probability of containing the depth.

Various methods are proposed to improve the 3D cost volume quality. P-MVSNet
(Luo et al. 2019) uses the isotropic and anisotropic 3D convolution to
estimate the depth map. AttMVS (Luo et al. 2020) applies attention mechanism
into the 3D cost volume to improve the robustness. UCS-MVSNet (Cheng et al.
2020) uses the uncertainty as a guide to adjust the 3D cost volume. EPP-MVSNet
(Ma et al. 2021) propose an epipolar-assembling module to enhance the 3D cost
volume. MVSTR (Zhu et al. 2021) relies on the 3D-geometry transformer
(Dosovitskiy et al. 2021) to obtain global context and 3D consistency. Also,
considering the occlusion, Vis-MVSNet (Zhang et al. 2020) and PVSNet (Xu and
Tao 2020c) introduce the visibility to filter out unreliable regions. Besides,
MVSCRF (Xue et al. 2019) uses a conditional random field to ensure the
smoothness of the depth map, and Uni-MVSNet (Peng et al. 2022) combines the
regression and classification by the unified focal loss.

Figure 2:  The pipeline of DispMVS. After extracting features from input
images, DispMVS first uses a random depth map to initialize the E-flow between
each pair and then triangulates a new depth map by the E-flow that is updated
through a GRU module. Finally, with several iterations, DispMVS can
reconstruct a high-quality depth map.

In addition, unsupervised MVS has also achieved impressive results. Un-MVSNet
(Khot et al. 2019) uses photometric as a guide to learning the depth map.
MVSNet2 (Dai et al. 2019) uses multi-view depth maps to filter out occlusion.
M3VSNet (Huang et al. 2021) uses the normal to ensure smoothness and improve
the depth map. JADAS (Xu et al. 2021) combines depth and semantics to improve
the depth map. RC-MVSNet (Chang et al. 2022) trains a NeRF (Mildenhall et al.
2020) to maintain rendering consistency to solve ambiguity correspondences
among views.

#### RNN Method

Instead of using the 3D CNN to process the 3D cost volume, the RNN-based
method uses the more efficient LSTM (Shi et al. 2015) or GRU (Cho et al.
2014). R-MVSNet (Yao et al. 2019) uses the GRU to regularize the cost volume
sequentially. RED-Net (Liu and Ji 2020) utilizes a 2D recurrent encoder-
decoder structure to process the cost volume based on the GRU module. D2HC-
RMVSNet (Yan et al. 2020) and AA-RMVSNet (Wei et al. 2021) proposes a hybrid
architecture that combines the LSTM and the UNet (Ronneberger, Fischer, and
Brox 2015). PatchMatchNet (Wang et al. 2021) introduces an end-to-end
PatchMatch (Bleyer, Rhemann, and Rother 2011) with adaptive propagation during
each iteration and achieves competitive performance with lower GPU memory.
Recently, IterMVS (Wang et al. 2022) uses RAFT (Teed and Deng 2020; Lipson,
Teed, and Deng 2021) as backbone and iterative updates the depth map based on
the GRU module.

##  3 Method

In this section, we introduce the details of the proposed method. The pipeline
of DispMVS is demonstrated in Fig. 2. Unlike other methods depending on the
depth range and differentiable homograph warping to build the 3D cost volume,
we use a network to match pixels along the epipolar line and triangulate the
depth. Therefore, we first discuss the relationship between the flow, the
depth, and the E-flow. Then, we extend the E-flow to multi-view and explain
the details of DispMVS and the loss function.

###  3.1 Flow and Depth

Given a reference view $r$ and a source view $s$ with their interior matrix
$K_{r},K_{s}$ and their relative exterior matrix $[R_{s},T_{s}]$, we define
$d_{r},d_{s}$ as the depth, and $\vec{f}_{r\to s},\vec{f}_{s\to r}$ as the
flow of each view. Assuming that the scene is static, we can convert depth to
flow and vice versa according to the multi-view geometry (Hartley and
Zisserman 2003).

Depth The depth describes the 3D shape of an image and can re-project a pixel
on the image plane to 3D space. Eq. 1 re-projects a pixel $p_{r}$ in $r$ to
$P_{p_{r}}$ in 3D by its depth $d_{r}(p_{r})$, in which $\tilde{p}_{r}$ is the
homogeneous representation of $p_{r}$ for computation efficiency. $P_{p_{r}}$
can also be projected to $s$ by Eq. 2.

$\displaystyle P_{p_{r}}$ $\displaystyle=d_{r}(p_{r})K_{r}^{-1}\tilde{p}_{r}$
(1) $\displaystyle p_{s}$ $\displaystyle\simeq K_{s}(R_{s}P_{p_{r}}+T_{s})$
(2)

Flow The flow describes the movement of pixels on the image plane between two
images. For a matched pixel pair $p_{r}$ in $r$ and $p_{s}$ in $s$, we
calculate the flow $\vec{f}_{p_{r\to s}}$ by Eq. 3. Generally, the flow does
not need to follow geometry constraints and has two degrees of freedom.

$\displaystyle\vec{f}_{r\to s}(p_{r})=p_{s}-p_{r}$ (3)

Depth to Flow Eq. 4 shows how to convert the the $d_{r}(p_{r})$ to the
$f_{r}(p_{r})$, where $\Rightarrow$ denotes the conversion. We first re-
project $p_{r}$ to $P_{p_{r}}$ by $d_{r}(p_{r})$ as Eq. 1 shows and then
project $P_{p_{r}}$ to $s$ by Eq. 2 to get the matched pixel $p_{s}$. Finally,
we can calculate $\vec{f}_{r\to s}(p_{r})$ by Eq. 3.

$\displaystyle d_{r}(p_{r})\Rightarrow P_{p_{r}}\Rightarrow
p_{s}\Rightarrow\vec{f}_{r\to s}(p_{r})$ (4)

Flow to Depth Although triangulation is s straightforward method to convert
$f_{r}$ to $d_{r}$ (Hartley and Zisserman 2003), it has to solve a not
differentiable homogeneous linear function. Considering this, we use a
differentiable closed-form solution to calculate the depth, even though it is
not optimal. Given $p_{r}$ and $f_{r\to s}(p_{r})$, we can determine $p_{s}$
by Eq. 3. Based on multi-view geometric consistency, we have the constrain in
Eq. 5:

$\displaystyle
d_{r}(p_{r})K_{r}^{-1}\tilde{p}_{r}=R_{s}d_{s}(p_{s})K_{s}^{-1}\tilde{p}_{s}+T_{s}$
(5)

Let $T_{s}=(t_{sx},t_{sy},t_{sz})^{T}$,
$K_{r}^{-1}\tilde{p}_{r}=(p_{rx},p_{ry},p_{rz})^{T}$ and
$R_{s}K_{s}^{-1}\tilde{p}_{s}=(p_{sx},p_{sy},p_{sz})^{T}$. We can calculate
$d_{r}(p_{r})$ by Eq. 8:

$\displaystyle\left\\{\begin{array}[]{@{}l@{\quad}l}d_{xr}(p_{r})&=(t_{sx}p_{sz}-t_{sz}p_{rx})/(p_{sx}p_{sz}-p_{sz}p_{rx})\\\\[3.0pt]
d_{yr}(p_{r})&=(t_{sy}p_{sz}-t_{sz}p_{ry})/(p_{sy}p_{sz}-p_{sz}p_{rx})\end{array}\right.$
(8)

Eq. 8 shows that there are two ways to compute the depth, namely
$d_{xr}(p_{r})$ and $d_{yr}(p_{r})$, since $\vec{f}_{r\to s}(p_{r})$ is a 2D
vector that provides flow in $x$ dimension $\vec{f}_{xr\to xs}(p_{r})$ and $y$
dimension $\vec{f}_{yr\to ys}(p_{r})$. Theoretically, $d_{xr}(p_{r})$ equals
$d_{yr}(p_{r})$. However, a smaller flow is not numerically stable and will
bring noise into the triangulation. Therefore we select the depth triangulated
by the larger flow by Eq. 11:

$\displaystyle
d_{r}(p_{r})=\left\\{\begin{array}[]{@{}l@{\quad}l}d_{xr}(p_{r})&\mbox{if
$|\vec{f}_{xr\to xs}(p_{r})|\geq|\vec{f}_{yr\to ys}(p_{r})|$}\\\\[3.0pt]
d_{yr}(p_{r})&\mbox{if $|\vec{f}_{xr\to xs}(p_{r})|<|\vec{f}_{yr\to
ys}(p_{r})|$}\end{array}\right.$ (11)

Figure 3:  Depth ($d_{p_{r}}$), flow ($\vec{f}_{r\to s}(p_{r})$) and E-flow
($e_{r\to s}(p_{r})$). We draw $p_{s}$ in $r$ to visualize the flow and
E-flow. $d_{p_{r}}$ is the depth of $p_{r}$. $\vec{f}_{r\to s}(p_{r})$ and
$e_{r\to s}(p_{r})$ describe the way $p_{r}$ moves. In static scenes,
$d_{p_{r}}$, $\vec{f}_{r\to s}(p_{r})$ and $e_{r\to s}(p_{r})$ are all
correlated and can be converted to each other based on multi-view geometry.

###  3.2 E-flow: the epipolar disparity flow

The flow describes the movement of a pixel on the image plane but does not
obey the epipolar geometry, which introduces ambiguity when triangulating the
depth. Therefore, we use the epipolar geometry to restrain the flow and define
the epipolar disparity flow (E-flow) by Eq. 12, where $\vec{e}_{dir}$ is the
normalized direction vector of the epipolar line, and $\cdot$ is the dot
product of vectors.

$\displaystyle e_{r\to s}(p_{r})=\vec{e}_{dir}(p_{r})\cdot(p_{s}-p_{r})$ (12)

E-flow and Flow Compared with the flow, the E-flow is a scalar and only moves
on the epipolar line. In the static scene, the flow and the E-flow are two
different ways to describe pixel movement, and their relationship is shown in
Eq. 13. Fig. 3 visualizes the flow $\vec{f}_{r\to s}(p_{r})$ and the E-flow
$e_{r\to s}(p_{r})$ of pixel $p_{r}$.

$\displaystyle\vec{f}_{r\to s}(p_{r})=\vec{e}_{dir}(p_{r})e_{r\to s}(pr)$ (13)

E-flow and Depth Considering the relationship between the E-flow and the flow,
and the relationship between the flow and the depth, we can convert E-flow to
depth, and vice versa by Eq. 14, in which $\Leftrightarrow$ denotes the
interconversion.

$\displaystyle e_{r\to s}\Leftrightarrow\vec{f}_{r\to s}\Leftrightarrow d_{r}$
(14)

Figure 4:  Different Sampling Spaces. (a) shows the sampling space of existing
methods that evenly samples in the 3D space to build a 3D cost volume. (b)
shows the sampling space of DispMVS that evenly samples on each source image
plane and estimates the depth by triangulation, whose distribution is decided
by the relative poses of the source image.

###  3.3 DispMVS

Given a reference image $r$ and $N$ source images $s_{i}(1<=i<=N)$, DispMVS
firstly extracts features from all input images and then iteratively updates
the depth from random initialization. DispMVS separately estimates the E-flow
of each pair by a 2D cost volume and converts the E-flow to the depth for
later multi-view depth fusion by a weighted sum. Fig. 2 shows the pipeline of
DispMVS with the first iteration at the coarse stage.

Feature Extraction Following RAFT (Teed and Deng 2020), we use two identical
encoders to extract features from input images. One encoder simultaneously
extracts matching features from input images to calculate similarities.
Meanwhile, another encoder extracts context features from the reference image
to iteratively update the E-flow. As we apply a coarse-to-fine strategy to
speed up efficiency and accuracy, these encoders use a UNet structure
(Ronneberger, Fischer, and Brox 2015) to extract coarse feature maps
$c_{r},c_{s_{i}}$ with resolution $1/16$ for the coarse stage and
$f_{r},f_{s_{i}}$ with resolution $1/4$ for the fine stage. Besides, we also
use the deformable convolutional network (Dai et al. 2017) at the decoder part
to capture valuable information.

Initialization of E-flow DispMVS relies on the E-flow to estimate the depth,
so it needs an initial depth as the starting point. We adopt the
initialization strategy from PatchMatch (Bleyer, Rhemann, and Rother 2011) and
initialize $d_{r,0}$ by Eq. 15, where $rand$ comes from a normal distribution,
and $(d_{min},d_{max})$ are lower and upper bound of the depth range. With
$d_{r,0}$, DispMVS can initialize the E-flow for pair by Eq. 14, which is zero
for the first iteration. Fig. 2 shows how DispMVS reconstructs a coarse depth
from a random depth.

$\displaystyle\frac{1}{d_{r,0}}={rand\times(\frac{1}{d_{min}}-\frac{1}{d_{max}})+\frac{1}{d_{max}}}$
(15)

E-flow Estimation After the initialization, DispMVS estimates $e_{s_{i}}$ of
each pair $(r,s_{i})$ by the GRU module. For $p_{r}$ and coarse feature
$c_{r}(p_{r})$, DispMVS uses $e_{r\to s_{i}}(p_{r})$ to find matched point
$p_{s_{i}}$ on $s_{i}$ and samples feature from $c_{s_{i}}(p_{s})$. To
increase the reception field, DispMVS applies $m_{s}$ times of the average-
pool on $c_{s_{i}}$ to generate features with different scales and evenly
samples $m_{p}$ points around $p_{s_{i},t}$ along the epipolar line at each
scale with a distance of one pixel. Then, DispMVS calculates the similarity
between $c_{r}(p_{r})$ and $m_{p}\times m_{s}$ features from $c_{s_{i}}$ by
Eq. 16 to build a 2D cost volume. In the end, DispMVS feeds the cost volume
and $e_{r\to s_{i}}(p_{r})$ to the GRU module to estimate a new $e_{r\to
s_{i}}(p_{r})$ and the weight $w_{s_{i}}(p_{r})$. In DispMVS, we set
$m_{s}=4,m_{p}=9$ at the coarse stage and $m_{s}=2,m_{p}=5$ at the fine stage.
Fig. 4 compares two different sampling spaces.

$\displaystyle
simi(c_{r}(p_{r}),c_{s_{i}}(p_{s_{i}}))=\sum_{0<=i<=D}c_{r}(p_{r})[i]c_{s_{i}}(p_{s_{i}})[i]$
(16)

Multi-View E-flow Estimation Since the E-flow only applies to two views,
DispMVS utilizes a weighted sum to extend the E-flow to the multi-view
situation. DispMVS converts $e_{r\to s_{i}}$ to $d_{r\to s_{i}}$ by Eq. 14 and
then fuses $d_{r}$ by Eq. 19, where $w_{s_{i}}$ is normalized by the softmax.
To be noted, DispMVS iteratively reconstruct the depth, which means that there
are several conversion between the depth and the E-flow, thus further
improving multi-view consistency.

$\displaystyle\left\\{\begin{array}[]{@{}l@{\quad}l}w_{s,t+1}&=softmax(w_{s,t+1})\\\\[3.0pt]
d_{r,t+1}&=\sum_{1<=i<=N}{d_{r\to s_{i},t+1}\times
w_{s_{i},t+1}}\end{array}\right.$ (19)

Coarse-to-Fine Stragety Following CasMVSNet (Gu et al. 2020), DispMVS uses a
coarse-to-fine strategy. DispMVS uses $c_{r},c_{s_{i}}$ with $t_{c}$
iterations at the coarse stage and $f_{r},f_{s_{i}}$ with $t_{f}$ iterations
at the fine stage. DispMVS starts from a random depth at the coarse stage and
upsamples (Teed and Deng 2020) the coarse depth to the fine stage for later
refinement. Generally, DispMVS needs more $t_{c}$ and fewer $t_{f}$ to improve
efficiency and accuracy.

Loss Function As DispMVS outputs a depth in each iteration, we calculate the
L1-loss for all depth maps to speed up convergence. To improve the stability
of the training procedure, we use the inverse depth range to normalize the
ground truth (gt) and the depth $d_{r,i}$. Eq. 20 shows our loss function, in
which $\gamma=0.9$:

$\displaystyle
loss=\sum_{j={t_{c},t_{f}}}{\sum_{0<=i<j}{\gamma^{i}|norm(gt_{i})-norm(d_{r,i})|}}$
(20)

##  4 Experiments

In this section, we benchmark our DispMVS on two public datasets and compare
it with a set of existing methods. We also conduct ablation experiments to
explore the effects of different settings of DispMVS.

Table 1: The evaluation results on DTUMVS (Aanæs et al. 2016). The lower the
Accuracy (Acc), Completeness (Comp), Overall and Mem (GPU Memory), the better.
We split methods into three categories and highlight the best in bold for
each.

| Method | Acc$\downarrow$ | Comp$\downarrow$ | Overall$\downarrow$ | Mem$\downarrow$
---|---|---|---|---|---
Trad | Comp (Campbell et al. 2008) | 0.835 | 0.554 | 0.695 | —
Furu (Furukawa and Ponce 2009) | 0.613 | 0.941 | 0.777 | —
Tola (Tola, Strecha, and Fua 2012) | 0.342 | 1.190 | 0.766 | —
Gipuma (Galliani et al. 2015) | 0.283 | 0.873 | 0.578 | —
Conv | MVSNet (Yao et al. 2018) | 0.396 | 0.527 | 0.462 | 9384M
Point-MVSNet (Chen et al. 2019) | 0.342 | 0.411 | 0.376 | —
P-MVSNet (Luo et al. 2019) | 0.406 | 0.434 | 0.420 | —
Fast-MVSNet (Yu and Gao 2020) | 0.336 | 0.403 | 0.370 | —
CVP-MVSNet (Yang et al. 2020) | 0.296 | 0.406 | 0.351 | —
Vis-MVSNet (Zhang et al. 2020) | 0.369 | 0.361 | 0.365 | 4775M
CIDER (Xu and Tao 2020a) | 0.417 | 0.437 | 0.427 | —
CasMVSNet (Gu et al. 2020) | 0.325 | 0.385 | 0.355 | 4591M
UCS-Net (Cheng et al. 2020) | 0.338 | 0.349 | 0.344 | 4057M
EPP-MVSNet (Ma et al. 2021) | 0.414 | 0.297 | 0.355 | —
UniMVSNet (Peng et al. 2022) | 0.352 | 0.278 | 0.315 | 3216M
GBiNet (Mi, Di, and Xu 2022) | 0.315 | 0.262 | 0.289 | 2018M
RNN | R-MVSNet (Yao et al. 2019) | 0.383 | 0.452 | 0.417 | —
D2HC-RMVSNet (Yan et al. 2020) | 0.395 | 0.378 | 0.386 | —
AA-RMVSNet (Wei et al. 2021) | 0.376 | 0.339 | 0.357 | 11973M
PatchMatchNet (Wang et al. 2021) | 0.427 | 0.277 | 0.352 | 1629M
IterMVS (Wang et al. 2022) | 0.373 | 0.354 | 0.363 | 842M
DispMVS (ours) | 0.354 | 0.324 | 0.339 | 1368M

Figure 5:  Point clouds We visualize some point clouds generated by DispMVS on
DTUMVS (Aanæs et al. 2016) and Tanks & Temple (Knapitsch et al. 2017).  Table
2:  The evaluation results on Tanks & Temple (Knapitsch et al. 2017). Higher
scores indicate higher quality of the point cloud. We split methods into two
categories and highlight the best in bold for each.

Advanced | Intermediate
---|---
Method | Mean | Aud. | Bal. | Cou. | Mus. | Pal. | Tem. | Mean | Fam. | Fra. | Hor. | Lig. | M60 | Pan. | Pla. | Tra
MVSNet (Yao et al. 2018) | - | - | - | - | - | - | - | 43.48 | 55.99 | 28.55 | 25.07 | 50.79 | 53.96 | 50.86 | 47.90 | 34.69
Point-MVSNet (Chen et al. 2019) | - | - | - | - | - | - | - | 48.27 | 61.79 | 41.15 | 34.20 | 50.79 | 51.97 | 50.85 | 52.38 | 43.06
UCS-Net (Cheng et al. 2020) | - | - | - | - | - | - | - | 54.83 | 76.09 | 53.16 | 43.03 | 54.00 | 55.60 | 51.49 | 57.38 | 47.89
Vis-MVSNet (Zhang et al. 2020) | 33.78 | 20.79 | 38.77 | 32.45 | 44.20 | 28.73 | 37.70 | 60.03 | 77.40 | 60.23 | 47.07 | 63.44 | 62.21 | 57.28 | 60.54 | 52.07
CasMVSNet (Gu et al. 2020) | 31.12 | 19.81 | 38.46 | 29.10 | 43.87 | 27.36 | 28.11 | 56.42 | 76.36 | 58.45 | 46.20 | 55.53 | 56.11 | 54.02 | 58.17 | 46.56
EPP-MVSNet (Ma et al. 2021) | 35.72 | 21.28 | 39.74 | 35.34 | 49.21 | 30.00 | 38.75 | 61.68 | 77.86 | 60.54 | 52.96 | 62.33 | 61.69 | 60.34 | 62.44 | 55.30
UniMVSNet (Peng et al. 2022) | 38.96 | 28.33 | 44.36 | 39.74 | 52.89 | 33.80 | 34.63 | 64.36 | 81.20 | 66.43 | 53.11 | 63.46 | 66.09 | 64.84 | 62.23 | 57.53
GBiNet (Mi, Di, and Xu 2022) | 37.32 | 29.77 | 42.12 | 36.30 | 47.69 | 31.11 | 36.93 | 61.42 | 79.77 | 67.69 | 51.81 | 61.25 | 60.37 | 55.87 | 60.67 | 53.89
R-MVSNet (Yao et al. 2019) | 24.91 | 12.55 | 29.09 | 25.06 | 38.68 | 19.14 | 24.96 | 48.40 | 69.96 | 46.65 | 32.59 | 42.95 | 51.88 | 48.80 | 52.00 | 42.38
D2HC-RMVSNet (Yan et al. 2020) | - | - | - | - | - | - | - | 59.20 | 74.69 | 56.04 | 49.42 | 60.08 | 59.81 | 59.61 | 60.04 | 53.92
AA-RMVSNet (Wei et al. 2021) | 33.53 | 20.96 | 40.15 | 32.05 | 46.01 | 29.28 | 32.71 | 61.51 | 77.77 | 59.53 | 51.53 | 64.02 | 64.05 | 59.47 | 60.85 | 54.90
PatchmatchNet (Wang et al. 2021) | 32.31 | 23.69 | 37.73 | 30.04 | 41.80 | 28.31 | 32.29 | 53.15 | 66.99 | 52.64 | 43.24 | 54.87 | 52.87 | 49.54 | 54.21 | 50.81
IterMVS (Wang et al. 2022) | 33.24 | 22.95 | 38.74 | 30.64 | 43.44 | 28.39 | 35.27 | 56.94 | 76.12 | 55.80 | 50.53 | 56.05 | 57.68 | 52.62 | 55.70 | 50.99
DispMVS (ours) | 34.90 | 26.09 | 38.01 | 33.19 | 44.90 | 28.49 | 38.75 | 59.07 | 74.73 | 60.67 | 54.13 | 59.58 | 58.02 | 53.39 | 58.63 | 53.42

#### Datasets

We use three different datasets throughout our experiments. The DTUMVS (Aanæs
et al. 2016) is an indoor dataset in a controlled environment containing 79
scenes for training, 22 for testing, and 18 for validation. The BlendedMVS
(Yao et al. 2020) is a large dataset captured from various outdoor scenes,
with 106 scenes for training and the rest 7 scenes for testing. The
Tanks&Temple (Knapitsch et al. 2017) is an outdoor multi-view stereo benchmark
that contains 14 real-world scenes under complex conditions.

#### Implementation

We implement DispMVS by PyTorch (Paszke et al. 2019) and train two models on
the DTUMVS and the BlendedMVS separately. On the DTUMVS, we set the image
resolution to $640\times 512$ and $N=5$. On the BlendedMVS, we set the image
resolution to $768\times 576$ and $N=5$. For all models, we apply the training
strategy in PatchmatchNet (Wang et al. 2021) for better learning of the weight
and use the Adam (Kingma and Ba 2015)( $\beta_{1}=0.9,\beta_{2}=0.999$ )
optimizer with an initial learning rate of 0.0002 that halves every four
epochs for 16 epochs. The training procedure is finished on two V100 with
$t_{c}=8,t_{f}=2$ considering the GPU memory limitation. During the
evaluation, we filter and fuse all depth maps into point clouds to compare
with the ground truth.

###  4.1 Evaluation on DTUMVS

We evaluate the DTUMVS on the test part and resize all images to $1600\times
1152$ with $N=5$. Table 1 compares DispMVS with other state-of-the-art
methods. We split existing methods into traditional methods, 3D convolution-
based methods, and RNN-based methods. DispMVS has the best overall score among
RNN-based methods and is 0.024 lower than IterMVS (Wang et al. 2022) and 0.018
lower than AA-RMVSNet (Wei et al. 2021). DispMVS ranks the 3rd among all
methods. GBiNet (Mi, Di, and Xu 2022) and UniMVSNet (Peng et al. 2022) are the
top two methods, but they incur much higher GPU memory. We visualize some
point clouds of DTUMVS generated by DispMVS in the first row of Fig. 5. These
qualitative and quantitative experimental results demonstrate the
effectiveness of DispMVS in obtaining depth by triangulation, even though
DispMVS does not construct any 3D cost volume.

###  4.2 Evaluation on Tanks&Tmple

As Tanks&Temple does not provide training samples, we apply the model
pretrained on the BlendedMVS to it. We resize all images of Tanks&Temple to
$1600\times 1152$ and set $N=7$. Table 2 shows the results of learning-based
methods, split into 3D convolution-based and RNN-based. Tanks&Temple contains
two subsets, Intermedia and Advanced. DispMVS achieves the best mean score on
the Advanced subset among the RNN-based method. Although AA-RMVSNet (Wei et
al. 2021) outperforms DispMVS on the Intermedia subset, AA-RMVSNet uses nearly
ten times more GPU memory than DispMVS. Overall, the results on Tanks&Temple
demonstrate that DispMVS has robust generalization with a low amount of GPU
memory. The second row of Fig. 5 shows point clouds generated on the
Tanks&Temple by DispMVS.

###  4.3 Ablation Studies

In this subsection, we discuss the core parts of our method. Considering that
the RAFT structure has been thoroughly studied in (Teed and Deng 2020; Lipson,
Teed, and Deng 2021; Wang et al. 2022), we conduct ablation experiments on the
coarse-to-fine strategy, the random initialization, and the changes in depth
range. Throughout all ablation experiments, we use the DTUMVS as a baseline
dataset.

The coarse-to-fine stragety DispMVS-M firstly estimates the depth with the
feature from $1/16$ and refines the depth with the feature from $1/4$.
DispMVS-S only extracts features from $1/8$ to recover the depth map to make
the comparison fairer. Table. 3 shows that the coarse-to-fine strategy
dramatically improves the overall score from 0.455 for DispMVS-M to 0.339 for
DispMVS-S. Therefore, we choose DispMVS-M as the method used in this paper.

Random Initilization Unlike existing methods that build the 3D cost volume
from a known depth range, DispMVS starts from a random initial depth, which
means that the input of DispMVS could be different every time. To measure the
effects of the random initialization, we conduct three times of initilization
without fixing the random seed and evaluate point clouds. Table 4 shows that
the variance of metrics between different inferences is smaller than 1e-6,
which proves that DispMVS are robust to the random initial depth and can
always generate a high-quality depth map.

Table 3: Comparison between single stage and multi stage. “-S” indicates reconstruction with only one resolution at $1/8$, while “-M” indicates reconstruction with multiple resolutions (the method in this paper). Method | Acc$\downarrow$ | Comp$\downarrow$ | Overall$\downarrow$
---|---|---|---
DispMVS-S | 0.500 | 0.410 | 0.455
DispMVS-M | 0.354 | 0.324 | 0.339

Depth Range The existing MVS methods split a given depth range into several
bins and build a 3D cost volume by differentiable homography. However, DispMVS
is insensitive to the depth range as it only constructs a 2D cost volume on
the image plane along the epipolar line. We select two state-of-the-art
methods (GBiNet (Mi, Di, and Xu 2022) and IterMVS (Wang et al. 2022)) and
manually change the depth range by Eq. 23. All methods are trained by the same
dataset with the same depth range. Table. 5 shows that performance of GBiNet
and IterMVS decreases dramatically, but DispMVS can be robust to these
changes. Fig. 1 visualizes point clouds generated by different methods with
different depth ranges, where GBiNet and IterMVS cannot converge when the
depth range is too large.

$\displaystyle
range_{x}=\left\\{\begin{array}[]{<EMAIL_ADDRESS>d_{max}&=d_{max}\times x\end{array}\right.\vspace{-1em}$ (23)

###  4.4 Limitations

As DispMVS needs to keep building the 2D cost volume during the iteration, its
computational efficiency is relatively low. In our experiment, DispMVS needs
around 0.7 seconds to process a view on the DTUMVS. Compared with IterMVS
(Wang et al. 2022) which only needs around 0.3 seconds per view, DispMVS needs
a more efficient epipolar matching module. In addition, DispMVS needs around
48G GPU memory during training because DispMVS needs several iterations to
update the depth by the GRU module, which needs to save all gradients and
intermediate results.

Table 4: Evaluate the random initialization. A lower variance means the difference between multi results is smaller. Method | Acc$\downarrow$ | Comp$\downarrow$ | Overall$\downarrow$
---|---|---|---
rand-1 | 0.353829 | 0.324110 | 0.338970
rand-2 | 0.354811 | 0.324946 | 0.339878
rand-3 | 0.354272 | 0.324324 | 0.339298
variance | 2.418e-7 | 1.890e-7 | 2.110e-7
Table 5:  Influences of changing the depth range. The lower, the better for
all metrics under different depth ranges.

Range | Method | Acc$\downarrow$ | Comp$\downarrow$ | Overall$\downarrow$
---|---|---|---|---
$range_{1}$ | GBiNet (Mi, Di, and Xu 2022) | 0.315 | 0.262 | 0.289
IterMVS (Wang et al. 2022) | 0.373 | 0.354 | 0.363
DispMVS (ours) | 0.354 | 0.324 | 0.339
$range_{2}$ | GBiNet (Mi, Di, and Xu 2022) | 0.480 | 0.556 | 0.518
IteMVS (Wang et al. 2022) | 0.532 | 1.471 | 1.002
DispMVS (ours) | 0.348 | 0.404 | 0.376
$range_{3}$ | GBiNet (Mi, Di, and Xu 2022) | 0.618 | 1.303 | 0.960
IterMVS (Wang et al. 2022) | 0.935 | 6.985 | 3.960
DispMVS (ours) | 0.314 | 0.671 | 0.493

##  5 Conclusion

This paper introduces a new pipeline of MVS, called DispMVS, which does not
need to build any 3D cost volumes but triangulates the depth map by multi-view
geometry. DispMVS is a depth range invariant method and can be generalized to
the dataset with different ranges with the training set, which proves that 3D
cost volume is unnecessary for MVS. Compared with existing learning-based
methods, DispMVS lets the network focus on matching pixels that the CNN
network is good at and uses the multi-view geometry to deal with the geometry
information. Experiments on datasets show that DispMVS achieves comparable
results with other 3D convolution methods and outperforms RNN-based methods
with a lower GPU memory requirement.

#### Acknowledgments

This work was supported in part by grant RMGS2021-8-10 from Hong Kong Research
Matching Grant Scheme and the NVIDIA Academic Hardware Grant.

Supplementary material

##  1 Depth Normalization

To make our method more numeric stable, we apply a depth normalization by Eq.
24, where we use the inversed minimum depth $d_{min}$ and inversed maximum
depth $d_{max}$ to normalize the input depth. The depth normalization can
decrease the effect of outliers, especially in the multi-view depth fusion and
the loss function. In the multi-view depth fusion, we convert the epipolar
disparity flow (E-flow) to depth by triangulation, which may introduce
outliers when E-flow is too small. As for the loss function, the normalization
can make it easier for the network to converge when the depth range is
different among datasets.

$\displaystyle
depth\\_norm=\frac{1/depth-1/depth_{max}}{1/depth_{min}-1/depth_{max}}$ (24)

##  2 GRU Update

To update the E-flow, we utilize two GRU modules at the coarse and fine stages
of DispMVS with the same network structure of RAFT (Teed and Deng 2020). The
input and output of this module is seen in Eq. 29, where $f_{t},f_{r},f_{h}$
are 2D convolution networks and $W_{t},W_{r},W_{h}$ are the corresponding
parameters. After updating E-flow, we triangulate the depth and fuse them in
each iteration. Fig. 6 shows the depth maps generated by DispMVS on DTU and
Tanks&Temple, in which we can see that the depth map recovers from coarse to
fine.

$\displaystyle\left\\{\begin{array}[]{@{}l@{\quad}l}z_{t}&=\sigma(f_{t}([h_{t-1},x_{t}],W_{t}))\\\\[3.0pt]
r_{t}&=\sigma(f_{r}([h_{t-1},x_{t}],W_{r}))\\\\[3.0pt]
\tilde{h_{t}}&=tanh(f_{h}([r_{t}\bigodot h_{t-1},x_{t}],W_{h}))\\\\[3.0pt]
h_{t}&=(1-z_{t})\bigodot h_{t-1}+z_{t}\bigodot\tilde{h_{t}}\end{array}\right.$
(29)

##  3 Depth Upsampling

Following RAFT (Teed and Deng 2020), we upsample a coarse depth map to a fine
depth map through a convex combination of each pixel’s nine neighbors instead
of directly using the bilinear upsampling. Since this upsampling module
acquires the fusion weights by learning, it can retain more details and have
sharp edges at the boundaries.

##  4 Depth Fusion

DispMVS only predicts the depth map and needs to fuse them to point clouds for
further evaluation. The fusion can filter the depth map to remove inconsistent
regions and reduce the noise in the point cloud. Following MVSNet (Yao et al.
2018), we use geometric consistency to filger out inconsistent regions and
reduce the noise in the point cloud. Geometric consistency contains two steps,
the first step projects depth maps of source images to the reference image to
mask out regions with large depth differences, and the second step reprojects
the depth map of the reference image to source images to mask out regions with
large pixel displacements. Besides, DispMVS also uses the depth range to
filter out points out of bound. After filtering out these inconsistent regions
and invalid regions, DispMVS reproject pixels to 3D space to generate point
clouds. Fig. 7 and Fig. 8 show point clouds generated by DispMVS.

Figure 6:  Depth Maps in each iteration on DTUMVS and Tanks&Temple DispMVS
iteratively recovers the depth map. An interesting phenomenon is that DispMVS
cannot reconstruct the sky region, where DispMVS cannot find reliable matches
and is far away from the view. But DispMVS can reconstruct the thin structure
nearby.

Figure 7:  Results on DTUMVS We visualize 22 scenes in the test.

Figure 8:  Results on Tanks&Temple We visualize all scenes in Intermedia and
Advanced.

## References

  * Aanæs et al. (2016) Aanæs, H.; Jensen, R. R.; Vogiatzis, G.; Tola, E.; and Dahl, A. B. 2016.  Large-scale data for multiple-view stereopsis.  _International Journal of Computer Vision_ , 120(2): 153–168.
  * Bleyer, Rhemann, and Rother (2011) Bleyer, M.; Rhemann, C.; and Rother, C. 2011.  Patchmatch stereo-stereo matching with slanted support windows.  In _Bmvc_ , volume 11, 1–11.
  * Campbell et al. (2008) Campbell, N. D.; Vogiatzis, G.; Hernández, C.; and Cipolla, R. 2008.  Using multiple hypotheses to improve depth-maps for multi-view stereo.  In _European Conference on Computer Vision_ , 766–779. Springer.
  * Chang et al. (2022) Chang, D.; Božič, A.; Zhang, T.; Yan, Q.; Chen, Y.; Süsstrunk, S.; and Nießner, M. 2022.  RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering.  In _European conference on computer vision_.
  * Chen et al. (2019) Chen, R.; Han, S.; Xu, J.; and Su, H. 2019.  Point-based multi-view stereo network.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1538–1547.
  * Cheng et al. (2020) Cheng, S.; Xu, Z.; Zhu, S.; Li, Z.; Li, L. E.; Ramamoorthi, R.; and Su, H. 2020\.  Deep stereo using adaptive thin volume representation with uncertainty awareness.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2524–2534.
  * Cho et al. (2014) Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , 1724–1734.
  * Dai et al. (2017) Dai, J.; Qi, H.; Xiong, Y.; Li, Y.; Zhang, G.; Hu, H.; and Wei, Y. 2017.  Deformable convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , 764–773.
  * Dai et al. (2019) Dai, Y.; Zhu, Z.; Rao, Z.; and Li, B. 2019.  Mvs2: Deep unsupervised multi-view stereo with multi-view symmetry.  In _2019 International Conference on 3D Vision (3DV)_ , 1–8. Ieee.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2021\.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_.
  * Furukawa and Ponce (2009) Furukawa, Y.; and Ponce, J. 2009.  Accurate, dense, and robust multiview stereopsis.  _IEEE transactions on pattern analysis and machine intelligence_ , 32(8): 1362–1376.
  * Galliani et al. (2015) Galliani, S.; et al. 2015.  Massively parallel multiview stereopsis by surface normal diffusion.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 873–881.
  * Goesele et al. (2007) Goesele, M.; Snavely, N.; Curless, B.; Hoppe, H.; and Seitz, S. M. 2007.  Multi-view stereo for community photo collections.  In _2007 IEEE 11th International Conference on Computer Vision_ , 1–8. IEEE.
  * Gu et al. (2020) Gu, X.; Fan, Z.; Zhu, S.; Dai, Z.; Tan, F.; and Tan, P. 2020.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2495–2504.
  * Hartley and Zisserman (2003) Hartley, R.; and Zisserman, A. 2003.  _Multiple view geometry in computer vision_.  Cambridge university press.
  * Huang et al. (2021) Huang, B.; Yi, H.; Huang, C.; He, Y.; Liu, J.; and Liu, X. 2021.  M3VSNet: Unsupervised multi-metric multi-view stereo network.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , 3163–3167. IEEE.
  * Khot et al. (2019) Khot, T.; Agrawal, S.; Tulsiani, S.; Mertz, C.; Lucey, S.; and Hebert, M. 2019.  Learning unsupervised multi-view stereopsis via robust photometric consistency.  _arXiv preprint arXiv:1905.02706_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _International Conference on Learning Representations_.
  * Knapitsch et al. (2017) Knapitsch, A.; Park, J.; Zhou, Q.-Y.; and Koltun, V. 2017.  Tanks and temples: Benchmarking large-scale scene reconstruction.  _ACM Transactions on Graphics (ToG)_ , 36(4): 1–13.
  * Kostrikov, Horbert, and Leibe (2014) Kostrikov, I.; Horbert, E.; and Leibe, B. 2014.  Probabilistic labeling cost for high-accuracy multi-view reconstruction.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , 1534–1541.
  * Kutulakos and Seitz (2000) Kutulakos, K. N.; and Seitz, S. M. 2000.  A theory of shape by space carving.  _International journal of computer vision_ , 38: 199–218.
  * Lhuillier and Quan (2005) Lhuillier, M.; and Quan, L. 2005.  A quasi-dense approach to surface reconstruction from uncalibrated images.  _IEEE transactions on pattern analysis and machine intelligence_ , 27: 418–433.
  * Lipson, Teed, and Deng (2021) Lipson, L.; Teed, Z.; and Deng, J. 2021.  Raft-stereo: Multilevel recurrent field transforms for stereo matching.  In _2021 International Conference on 3D Vision (3DV)_ , 218–227. IEEE.
  * Liu and Ji (2020) Liu, J.; and Ji, S. 2020.  A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 6050–6059.
  * Luo et al. (2019) Luo, K.; Guan, T.; Ju, L.; Huang, H.; and Luo, Y. 2019.  P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10452–10461.
  * Luo et al. (2020) Luo, K.; Guan, T.; Ju, L.; Wang, Y.; Chen, Z.; and Luo, Y. 2020.  Attention-aware multi-view stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1590–1599.
  * Ma et al. (2021) Ma, X.; Gong, Y.; Wang, Q.; Huang, J.; Chen, L.; and Yu, F. 2021.  EPP-MVSNet: Epipolar-assembling based Depth Prediction for Multi-view Stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5732–5740.
  * Merrell et al. (2007) Merrell, P.; Akbarzadeh, A.; Wang, L.; Mordohai, P.; Frahm, J.-M.; Yang, R.; Nistér, D.; and Pollefeys, M. 2007.  Real-time visibility-based fusion of depth maps.  In _2007 IEEE 11th International Conference on Computer Vision_ , 1–8. Ieee.
  * Mi, Di, and Xu (2022) Mi, Z.; Di, C.; and Xu, D. 2022.  Generalized Binary Search Network for Highly-Efficient Multi-View Stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12991–13000.
  * Mildenhall et al. (2020) Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2020.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European conference on computer vision_ , 405–421. Springer.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Peng et al. (2022) Peng, R.; Wang, R.; Wang, Z.; Lai, Y.; and Wang, R. 2022.  Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8645–8654.
  * Romanoni and Matteucci (2019) Romanoni, A.; and Matteucci, M. 2019.  Tapa-mvs: Textureless-aware patchmatch multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 10413–10422.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , 234–241. Springer.
  * Schönberger et al. (2016) Schönberger, J. L.; Zheng, E.; Frahm, J.-M.; and Pollefeys, M. 2016.  Pixelwise view selection for unstructured multi-view stereo.  In _European conference on computer vision_ , 501–518. Springer.
  * Seitz and Dyer (1999) Seitz, S. M.; and Dyer, C. R. 1999.  Photorealistic scene reconstruction by voxel coloring.  _International Journal of Computer Vision_ , 35: 151–173.
  * Shen (2013) Shen, S. 2013.  Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes.  _IEEE transactions on image processing_ , 22(5): 1901–1914.
  * Shi et al. (2015) Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; and Woo, W.-c. 2015.  Convolutional LSTM network: A machine learning approach for precipitation nowcasting.  _Advances in neural information processing systems_ , 28.
  * Teed and Deng (2020) Teed, Z.; and Deng, J. 2020.  Raft: Recurrent all-pairs field transforms for optical flow.  In _European conference on computer vision_ , 402–419. Springer.
  * Tola, Strecha, and Fua (2012) Tola, E.; Strecha, C.; and Fua, P. 2012.  Efficient large-scale multi-view stereo for ultra high-resolution image sets.  _Machine Vision and Applications_ , 23(5): 903–920.
  * Wang et al. (2021) Wang, F.; Galliani, S.; Vogel, C.; Speciale, P.; and Pollefeys, M. 2021.  Patchmatchnet: Learned multi-view patchmatch stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14194–14203.
  * Wang et al. (2022) Wang, F.; et al. 2022.  IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8606–8615.
  * Wang et al. (2020) Wang, Y.; Guan, T.; Chen, Z.; Luo, Y.; Luo, K.; and Ju, L. 2020.  Mesh-guided multi-view stereo with pyramid architecture.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2039–2048.
  * Wei et al. (2021) Wei, Z.; Zhu, Q.; Min, C.; Chen, Y.; and Wang, G. 2021.  Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6187–6196.
  * Xu et al. (2021) Xu, H.; Zhou, Z.; Qiao, Y.; Kang, W.; and Wu, Q. 2021.  Self-supervised multi-view stereo via effective co-segmentation and data-augmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 3030–3038.
  * Xu and Tao (2019) Xu, Q.; and Tao, W. 2019.  Multi-scale geometric consistency guided multi-view stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5483–5492.
  * Xu and Tao (2020a) Xu, Q.; and Tao, W. 2020a.  Learning inverse depth regression for multi-view stereo with correlation cost volume.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 12508–12515.
  * Xu and Tao (2020b) Xu, Q.; and Tao, W. 2020b.  Planar prior assisted patchmatch multi-view stereo.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 12516–12523.
  * Xu and Tao (2020c) Xu, Q.; and Tao, W. 2020c.  Pvsnet: Pixelwise visibility-aware multi-view stereo network.  _arXiv preprint arXiv:2007.07714_.
  * Xu et al. (2020) Xu, Z.; Liu, Y.; Shi, X.; Wang, Y.; and Zheng, Y. 2020.  Marmvs: Matching ambiguity reduced multiple view stereo for efficient large scale scene reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5981–5990.
  * Xue et al. (2019) Xue, Y.; Chen, J.; Wan, W.; Huang, Y.; Yu, C.; Li, T.; and Bao, J. 2019.  Mvscrf: Learning multi-view stereo with conditional random fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 4312–4321.
  * Yan et al. (2020) Yan, J.; Wei, Z.; Yi, H.; Ding, M.; Zhang, R.; Chen, Y.; Wang, G.; and Tai, Y.-W. 2020.  Dense hybrid recurrent multi-view stereo net with dynamic consistency checking.  In _European conference on computer vision_ , 674–689. Springer.
  * Yang et al. (2020) Yang, J.; Mao, W.; Alvarez, J. M.; and Liu, M. 2020.  Cost volume pyramid based depth inference for multi-view stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4877–4886.
  * Yang et al. (2022) Yang, Z.; Ren, Z.; Shan, Q.; and Huang, Q. 2022.  Mvs2d: Efficient multi-view stereo via attention-driven 2d convolutions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8574–8584.
  * Yao et al. (2018) Yao, Y.; Luo, Z.; Li, S.; Fang, T.; and Quan, L. 2018.  Mvsnet: Depth inference for unstructured multi-view stereo.  In _Proceedings of the European conference on computer vision (ECCV)_ , 767–783.
  * Yao et al. (2019) Yao, Y.; Luo, Z.; Li, S.; Shen, T.; Fang, T.; and Quan, L. 2019.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 5525–5534.
  * Yao et al. (2020) Yao, Y.; Luo, Z.; Li, S.; Zhang, J.; Ren, Y.; Zhou, L.; Fang, T.; and Quan, L. 2020\.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1790–1799.
  * Yu and Gao (2020) Yu, Z.; and Gao, S. 2020.  Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1949–1958.
  * Zhang et al. (2020) Zhang, J.; Yao, Y.; Li, S.; Luo, Z.; and Fang, T. 2020.  Visibility-aware Multi-view Stereo Network.  In _British Machine Vision Conference_.
  * Zheng et al. (2014) Zheng, E.; Dunn, E.; Jojic, V.; and Frahm, J.-M. 2014.  Patchmatch based joint view selection and depthmap estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1510–1517.
  * Zhu et al. (2021) Zhu, J.; Peng, B.; Li, W.; Shen, H.; Zhang, Z.; and Lei, J. 2021.  Multi-View Stereo with Transformer.  _arXiv preprint arXiv:2112.00336_.