# CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo

Weitao Chen111These authors contributed equally to this work.1 Hongbin
Xu111These authors contributed equally to this work.1,2 Zhipeng Zhou1 Yang
Liu1 Baigui Sun222Corresponding authors.1 Wenxiong Kang222Corresponding
authors.2 Xuansong Xie1 1Alibaba Group
2South China University of Technology
{hillskyxm<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The core of Multi-view Stereo(MVS) is the matching process among reference and
source pixels. Cost aggregation plays a significant role in this process,
while previous methods focus on handling it via CNNs. This may inherit the
natural limitation of CNNs that fail to discriminate repetitive or incorrect
matches due to limited local receptive fields. To handle the issue, we aim to
involve Transformer into cost aggregation. However, another problem may occur
due to the quadratically growing computational complexity caused by
Transformer, resulting in memory overflow and inference latency. In this
paper, we overcome these limits with an efficient Transformer-based cost
aggregation network, namely CostFormer. The Residual Depth-Aware Cost
Transformer(RDACT) is proposed to aggregate long-range features on cost volume
via self-attention mechanisms along the depth and spatial dimensions.
Furthermore, Residual Regression Transformer(RRT) is proposed to enhance
spatial attention. The proposed method is a universal plug-in to improve
learning-based MVS methods.

##  1 Introduction

Given a series of calibrated images from different views in one scene, Multi-
view Stereo (MVS) aims to recover the 3D information of the observed scene. It
is a fundamental problem in computer vision and widely applied to robot
navigation, autonomous driving, augmented reality, and etc. Recent learning-
based MVS networks Yao et al. (2018); Gu et al. (2020); Wang et al. (2021b)
have achieved inspiring success both in the quality and the efficiency of 3D
reconstruction. Generally, deep MVS approaches consist of the following five
steps: feature extraction from multi-view images via CNN network with shared
weights, differentiable warping to align all source features to the reference
view, matching cost computation from reference features and aligned source
features, matching cost aggregation or regularization, depth or disparity
regression.

Figure 1: Comparison with state-of-the-art MVS methods on DTU. Relationship
between error, GPU memory and run-time with image size 1152×864.

Current progresses in learning-based MVS primarily concentrate on the
limitation of reconstruction quality Wei et al. (2021); Yang et al. (2020a),
memory consumption Yan et al. (2020); Wei et al. (2021), and efficiency Wang
et al. (2021b, a). The basic network architecture of these works is based on
the pioneering backbone network called MVSNet Yao et al. (2018), which
provides an elegant and stable baseline. However, instead of taking the
inheritance of network design principle in MVSNet Yao et al. (2018) for
granted, we can rethink the task of MVS problem as a dense correspondence
problem Hosni et al. (2012) alternatively. The core of MVS is a dense
pixelwise correspondence estimation problem that searches the corresponding
pixel of a specific pixel in the reference image along the epipolar line in
all warped source images. No matter which task this correspondence estimation
problem is applied to, the matching task can be boiled down to a classical
matching pipeline Scharstein and Szeliski (2002): (1) feature extraction, and
(2) cost aggregation. In learning-based MVS methods, the transition from
traditional hand-crafted features to CNN-based features inherently solves the
former step of the classical matching pipeline via providing powerful feature
representation learned from large-scale data. However, _handling the cost
aggregation step by matching similarities between features without any prior
usually suffers from the challenges due to ambiguities generated by repetitive
patterns or background clutters_ Cho et al. (2021). Consequently, a typical
solution in MVSNet and its variants Yao et al. (2018); Gu et al. (2020); Wang
et al. (2021b) is to apply a 3D CNN or an RNN to regularize the cost volume
among reference and source views, rather than directly rely on the quality of
the initial correlation clues in cost volume. Although formulated variously in
previous methods, these methods either use hand-crafted techniques that are
agnostic to severe deformations or inherit the limitation of CNNs, e.g.
limited receptive fields, unable to discriminate incorrect matches that are
locally consistent.

In this work, we focus on the _cost aggregation step_ of cost volume and
propose a novel cost aggregation Transformer (CostFormer) to tackle the issues
above. Our CostFormer is based on Transformer Vaswani et al. (2017), which is
renowned for its global receptive field and long-range dependent
representation. By aggregating the matching cost in the cost volume, our
aggregation network can explore global correspondences and refine the
ambiguous matching points effectively with the help of the self-attention (SA)
mechanism in Transformer. Though the promising performances of Vision
Transformers have been proven in many applications Dosovitskiy et al. (2020);
Sun et al. (2021), the time and memory complexity of the key-query dot product
interaction in conventional SA grow quadratically with the spatial resolution
of inputs. Hence, replacing 3D CNN with Transformer may result in unexpected
extra occupancy in memory and latency in inference. Inspired by Wang et al.
(2021b), we further introduce the Transformer architecture into an iterative
multi-scale learnable PatchMatch pipeline. It inherits the advantages of the
long-range receptive field in Transformers, improving the reconstruction
performance substantially. Meantime, it also maintains a balanced trade-off
between efficiency and performance, which is competitive in the inference
speed and parameters magnitude compared with other methods.

Our main contributions are as follows:

(1) In this paper, we propose a novel Transformer-based cost aggregation
network called CostFormer, which can be plugged into learning-based MVS
methods to improve cost volume effectively. (2) CostFormer applies an
efficient Residual Depth-Aware Cost Transformer to cost volume, extending 2D
spatial attention to 3D depth and spatial attention. (3) CostFormer applies an
efficient Residual Regression Transformer between cost aggregation and depth
regression, keeping spatial attention. (4) The proposed CostFormer brings
benefits to learning-based MVS methods when evaluating DTU Aanæs et al.
(2016), Tanks $\&$ Temples Knapitsch et al. (2017) ETH3D Schöps et al. (2017)
and BlendedMVS Yao et al. (2020) datasets.

##  2 Related Work

###  2.1 Learning-based MVS Methods

Powered by the great success of deep learning-based techniques, many learning-
based methods have been proposed to boost the performance of Multi-view
Stereo. MVSNet Yao et al. (2018) is a landmark for the end-to-end network that
infers the depth map on each reference view for the MVS task. Feature maps
extracted by a 2D CNN on each view are reprojected to the same reference view
to build a variance-based cost volume. A 3D CNN is further used to regress the
depth map. Following this pioneering work, lots of efforts have been devoted
to boosting speed and reducing memory occupation. To relieve the burden of
huge memory cost, recurrent neural networks are utilized to regularize the
cost volume in AA-RMVSNet Wei et al. (2021). Following a coarse-to-fine manner
to develop a computationally efficient network, a recent strand of works
divide the single cost volume into several cost volumes at multiple stages,
like CasMVSNet Gu et al. (2020), CVP-MVSNet Yang et al. (2020a), UCSNet Cheng
et al. (2020), and etc. Inspired by the traditional PatchMatch stereo
algorithm, PatchMatchNet Wang et al. (2021b) inherits the pipeline in
PatchMatch stereo in an iterative manner and extend it into a learning-based
end-to-end network.

Figure 2: Structure of CostFormer based on PatchMatchNet.

###  2.2 Vision Transformer

The success of Transformer Vaswani et al. (2017) and its variants Dosovitskiy
et al. (2020); Liu et al. (2021) have motivated the development of Neural
Language Processing in recent years. Borrowing inspiration from these works,
Transformer has been successfully extended to vision tasks and proven to boost
the performance of image classification Dosovitskiy et al. (2020). Following
the pioneering work, many efforts are devoted to boosting the development of
various vision tasks with the powerful representation ability of Transformer.

In Li et al. (2021), the application of Transformer in the classic stereo
disparity estimation task is investigated thoughtfully. Swin Transformer Liu
et al. (2021) involves the hierarchical structure into Vision Transformers and
computes the representation with shifted windows. Considering Transformer’s
superiority in extracting global content information via attention mechanism,
many works attempt to utilize it in the task of feature matching. Given a pair
of images, CATs Cho et al. (2021) explore global consensus among correlation
maps extracted from a Transformer, which can fully leverage the self-attention
mechanism and model long-range dependencies among pixels. LoFTR Sun et al.
(2021) also leverages Transformers with a coarse-to-fine manner to model dense
correspondence. STTR Li et al. (2021) extends the feature matching Transformer
architecture to the task of stereo depth estimation task in a sequence-to-
sequence matching perspective. TransMVSNet Ding et al. (2021) is the most
relevant concurrent work compared with ours, which utilizes a Feature Matching
Transformer (FMT) to leverage self-attention and cross-attention to aggregate
long-range context information within and across images. Specifically, the
focus of TransMVSNet is on the enhancement of feature extraction before cost
aggregation, while our proposed CostFormer aims to improve the cost
aggregation process on cost volume.

##  3 Methodology

In this section, we introduce the detailed architecture of the proposed
CostFormer which focuses on the cost aggregation step of cost volume.
CostFormer contains two specially designed modules called Residual-Depth Aware
Cost Transformer (RDACT) and Residual Regression Transformer (RRT), which are
utilized to explore the relation between pixels within a long range and the
relation between different depth hypotheses during the evaluation process. In
Section Preliminary, we give a brief preliminary on the pipeline of our
method. Then we show the construction of RDACT and RRT respectively. Finally,
we show experiments.

Figure 3: Comparison of different methods on the DTU evaluation set. The
backbone of CostFormer is PatchMatchNet here.

###  3.1 Preliminary

In general, the proposed RDACT and RRT can be integrated with arbitrary cost
volume of learning-based MVS networks. Based on the patch match architecture
Wang et al. (2021b), we further explore the issue of cost aggregation on cost
volume. As shown in Figure 2, CostFormer based on PatchMatchNet Wang et al.
(2021b) extracts feature maps from multi-view images and performs
initialization and propagation to warp the features maps in source views to
reference view. Given a pixel $p$ at the reference view and its corresponding
pixel $p_{i,j}$ at the $i$-th source view under the $j$-th depth hypothesis
$d_{j}$ is defined as:

$p_{i,j}=K_{i}\cdot(R_{0,i}\cdot(K_{0}^{-1}\cdot p\cdot d_{j})+t_{0,i})$ (1)

where $R_{0,i}$ and $t_{0,i}$ denote the rotation and translation between the
reference view and $i$-th source view. $K_{0}$ and $K_{i}$ are the intrinsic
matrices of the reference and $i$-th source view. The warped feature maps at
the $i$-th source view $F_{i}(p_{i,j})$ are bilinearly interpolated to remain
the original resolution. Then, a cost volume is constructed from the
similarity of feature maps, and 3D CNNs are applied to regularize the cost
volume. Warped features from all source views are integrated into a single
cost for each pixel $p$ and depth hypothesis $d_{j}$ by computing the cost per
hypothesis $S_{i}(p,j)^{g}$ via group-wise correction as follows:

$S_{i}(p,j)^{g}=\frac{G}{C}<F_{0}(p)^{g},F_{i}(p_{i,j})^{g}>\in\mathbb{R}^{G}$
(2)

where $G$ is the group number, $C$ is the channel number, $<\cdot,\cdot>$ is
the inner product, $F_{0}(p)^{g}$ and $F_{i}(p_{i,j})^{g}$ are grouped
reference feature map and grouped source feature map at the $i$-th view
respectively. Then they aggregate over the views with a pixel-wise view weight
$w_{i}(p)$ to get $\overline{S}(p,j)$.

Taking no account of Transformer at the cost aggregation (CA) step, a CA
module firstly utilizes a small network with 3D convolution with $1\times
1\times 1$ kernels to obtain a single cost, $\mathcal{C}$ $\in R^{H\times
W\times D}$. For a spatial window of $K_{e}$ pixels
$\\{p_{k}\\}_{k=1}^{K_{e}}$ can be organized as a grid, per pixel additional
offsets $\\{\Delta p_{k}\\}_{k=1}^{K_{e}}$ can be learned for spatial
adaptation. The aggregated spatial cost $\widetilde{\mathcal{C}}(p,j)$ is
defined as:

$\widetilde{\mathcal{C}}(p,j)=\frac{1}{\sum_{k=1}^{K_{e}}w_{k}d_{k}}\overset{K_{e}}{\underset{k=1}{\sum}}w_{k}d_{k}\mathcal{C}(p+p_{k}+\Delta
p_{k},j)$ (3)

where $w_{k}$ and $d_{k}$ weight the cost $\mathcal{C}$ based on feature and
depth similarity. Given the sampling positions
${(p+p_{k}+\Delta{p_{k}})}_{k=1}^{K_{e}}$, corresponding features from $F_{0}$
are extracted via bilinear interpolation. Then group-wise correlation is
applied between the features at each sampling location and p. The results are
concatenated into a volume on which 3D convolution layers with 1×1×1 kernels
and sigmoid non-linearities are applied to output normalized weights
${\\{w_{k}\\}}_{k=1}^{K_{e}}$. The absolute difference in inverse depth
between each sampling point and pixel p with their j-th hypotheses are
collected. Then a sigmoid function on the inverted differences is applied to
obtain ${\\{d_{k}\\}}_{k=1}^{K_{e}}$.

The remarkable thing is that such cost aggregation inevitably suffers from
challenges due to ambiguities generated by repetitive patterns or background
clutters. The local mechanisms in ambiguities exist in many operations, such
as local propagation and spatial adaptation by small learnable slight offset.
CostFormer significantly alleviates these problems through RDACT and RRT. The
original CA module is also repositioned between RDACT and RRT.

After RRT, soft argmin is applied to get the regressed depth. Finally, a depth
refinement module is designed to refine the depth regression.

For CascadeMVS and other cascade architectures, CostFormer can be plugged into
similarly.

###  3.2 Residual Depth-Aware Cost Transformer

In this section, we explore the details of the Residual Depth-Aware Cost
Transformer (RDACT). Each RDACT consists of two parts. The first part is a
stack of Depth-Aware Transformer layer (DATL) and Depth-Aware Shifted
Transformer layer (DASTL), which deal with the cost volumes to explore the
relations sufficiently. The second part is the Re-Embedding Cost layer (REC)
which recovers the cost volume from the first part.

Given a cost volume $\mathcal{C}_{0}\in\mathbb{R}^{H\times W\times D\times
G}$, temporary intermediate cost volumes
$\mathcal{C}_{1}$,$\mathcal{C}_{2}$,…,$\mathcal{C}_{L}\in\mathbb{R}^{H\times
W\times D\times E}$ are firstly extracted by DATL and DASTL alternatively:

$\mathcal{C}_{k}=\text{DASTL}_{k}(\text{DATL}_{k}(\mathcal{C}_{k-1})),k=1,2,...,L$
(4)

where $\text{DATL}_{k}$ is the $k$-th Depth-Aware Transformer layer with
regular windows, $\text{DASTL}_{k}$ is the $k$-th Depth-Aware Transformer
layer with shifted windows, $E$ is the embedding dimension number of
$\text{DATL}_{k}$ and $\text{DASTL}_{k}$.

Then a Re-Embedding Cost layer is applied to the last $\mathcal{C}_{k}$,
namely $\mathcal{C}_{L}$, to recover $G$ from $E$. The output of RDACT is
formulated as:

$\mathcal{C}_{out}=\text{REC}(\mathcal{C}_{L})+\mathcal{C}_{0}$ (5)

where REC is the Re-Embedding Cost layer, and it can be a 3D convolution with
$G$ output channels. If $E=G$, $\mathcal{C}_{out}$ can be simply formulated
as:

$\mathcal{C}_{out}=\mathcal{C}_{L}+\mathcal{C}_{0}$ (6)

This residual connection allows the aggregation of different levels of cost
volumes; $\mathcal{C}_{out}$ instead of $\mathcal{C}_{0}$ is then aggregated
by the original aggregation network described in section 3.1. The whole RDACT
is shown in the red window in Figure 2.

Figure 4:  Comparison of different methods on Tanks&Temples. The Recall
reported by official benchmark is presented.

Before introducing the construction of DATL and DASTL, we dive into the
details of core constitutions called Depth-Aware Multi-Head Self-Attention
(DA-MSA) and Depth-Aware Shifted Multi-Head Self-Attention (DAS-MSA). Both DA-
MSA and DAS-MSA are based on Depth-Aware Self-Attention Mechanism. In order to
explain Depth-Aware Self-Attention Mechanism, we supply the knowledge about
Depth-Aware Patch Embedding and Depth-Aware Windows as preliminary.

Depth-Aware Patch Embedding: Obviously, directly applying the attention
mechanism for feature maps at pixel-wise level is quite costly in terms of GPU
memory usage. In order to tackle this issue, we propose a Depth-Aware Patch
Embedding to reduce the high memory cost and get an additional regularization.
Specifically, given a grouped cost volume before aggregation $\mathcal{C}$
$\in\mathbb{R}^{H\times W\times D\times G}$, a depth-aware patch embedding is
firstly applied to $\mathcal{C}$ to get tokens. It consists of a 3D
convolution with kernel size $h\times w\times d$ and a layer normalization.To
downsample the spatial sizes of cost volume and keep the depth hypotheses, we
set $h$ and $w$ to more than 1 and $d$ as 1. So the sample ratio is adaptive
for memory cost and run time. Before convolution, cost volume will be padded
to fit the spatial sizes and downsampling ratio. After layer
normalization(LN), these embedded patches are further partitioned by depth-
aware windows.

Depth-Aware Windows: Beyond the nonlinear and linear global self-attention,
local self-attention within a window has been proven to be more effective and
efficient. As an example of 2D windows, Swin Transformer Liu et al. (2021)
directly applies multi-head self-attention mechanisms on non-overlapping 2D
windows to avoid the big computation complexity of global tokens. Extended
from the 2D spatial window, an embedded cost volume patch
$\in\mathbb{R}^{H^{*}\times W^{*}\times D^{*}\times G}$ with depth information
is partitioned into non-overlapping 3D windows. These local windows are then
transposed and reshaped to local cost tokens. Assuming the sizes of these
windows are $hs\times ws\times ds$, the total number of tokens is
$\lceil{\frac{H^{*}}{h_{s}}}\rceil\times\lceil{\frac{W^{*}}{w_{s}}}\rceil\times\lceil{\frac{D^{*}}{d_{s}}}\rceil$.
These local tokens are further processed by the multi-head self-attention
mechanism.

Depth-Aware Self-Attention Mechanism: For a cost window token
$X\in\mathbb{R}^{h_{s}\times w_{s}\times d_{s}\times G}$, the query, key, and
value matrices $Q$, $K$ and $V\in\mathbb{R}^{h_{s}\times w_{s}\times
d_{s}\times G}$ are computed as:

$Q=XP_{Q},K=XP_{K},V=XP_{V}$ (7)

where $P_{Q}$, $P_{K}$, and $P_{V}\in\mathbb{R}^{G\times G}$ are projection
matrices shared across different windows. By introducing depth and spatial
aware relative position bias $B1\in\mathbb{R}^{(h_{s}\times
h_{s})\times(w_{s}\times w_{s})\times(d_{s}\times d_{s})}$ for each head, the
depth-aware self-attention(DA-SA1) matrix within a 3D local window is thus
computed as:

$\text{DA-SA1}=Attention1(Q1,K1,V1)=SoftMax(\frac{Q1K1^{T}}{\sqrt{G}}+B1)V1$
(8)

Where $Q1$, $K1$ and $V1\in\mathbb{R}^{h_{s}w_{s}d_{s}\times G}$ are reshaped
from $Q$, $K$ and $V\in\mathbb{R}^{h_{s}\times w_{s}\times d_{s}\times G}$.
The process of DATL with LayerNorm(LN) and multi-head DA-SA1 at the current
level is formulated as:

$\widehat{X}^{l}=\text{DA-MSA1}((\text{LN}(X^{l-1}))+X^{l-1}$ (9)

By introducing depth-aware relative position bias
$B2\in\mathbb{R}^{d_{s}\times d_{s}}$ for each head, the depth-aware self-
attention(DA-SA2) matrix along the depth dimension is an alternative module to
DATL and thus computed as:

$\text{DA-SA2}=Attention2(Q2,K2,V2)=SoftMax(\frac{Q2K2^{T}}{\sqrt{G}}+B2)V2$
(10)

Where $Q2$, $K2$ and $V2\in\mathbb{R}^{h_{s}w_{s}\times d_{s}\times G}$ are
reshaped from $Q$, $K$ and $V\in\mathbb{R}^{h_{s}\times w_{s}\times
d_{s}\times G}$. $B1$ and $B2$ will be along the depth dimension and lie in
the range of $[-d_{s}+1,d_{s}-1]$. Along the height and width dimension, $B1$
lies in the range of $[-h_{s}+1,h_{s}-1]$ and $[-w_{s}+1,w_{s}-1]$. In
practice, we parameterize a smaller-sized bias matrix
$\overline{B1}\in\mathbb{R}^{(2h_{s}-1)\times(2w_{s}-1)\times(2d_{s}-1)}$ from
$B1$ and perform the attention functionfor $f$ times in parallel, and then
concatenate the depth-aware multi-head self-attention (DA-MSA) outputs. The
process of DATL with LayerNorm(LN), multi-head DA-SA1, and DA-SA2 at the
current level is formulated as:

$\widehat{X}^{l}=\text{DA-MSA1}(\text{LN}(\text{DA-
MSA2}(\text{LN}(X^{l-1}))))+X^{l-1}$ (11)

Then, an MLP module that has two fully-connected layers with GELU non-
linearity between them is used for further feature transformations:

$X^{l}=\text{MLP}(\text{LN}(\widehat{X}^{l})))+\widehat{X}^{l}$ (12)

Compared with global attention, local attention makes it possible for
computation in high resolution.

Methods |  Intermediate Group (F-score $\uparrow$) |  Advanced Group (F-score $\uparrow$)
---|---|---
Mean | Fam. | Fra. | Hor. | Lig. | M60 | Pan. | Pla. | Tra. | Mean | Aud. | Bal. | Cou. | Mus. | Pal. | Tem.
MVSNet Yao et al. (2018) | 43.48 | 55.99 | 28.55 | 25.07 | 50.79 | 53.96 | 50.86 | 47.90 | 34.69 | - | - | - | - | - | - | -
CasMVSNet Gu et al. (2020) | 56.84 | 76.37 | 58.45 | 46.26 | 55.81 | 56.11 | 54.06 | 58.18 | 49.51 | 31.12 | 19.81 | 38.46 | 29.10 | 43.87 | 27.36 | 28.11
UCS-Net Cheng et al. (2020) | 54.83 | 76.09 | 53.16 | 43.03 | 54.00 | 55.60 | 51.49 | 57.38 | 47.89 | - | - | - | - | - | - | -
CVP-MVSNet Yang et al. (2020b) | 54.03 | 76.50 | 47.74 | 36.34 | 55.12 | 57.28 | 54.28 | 57.43 | 47.54 | - | - | - | - | - | - | -
PVA-MVSNet Yi et al. (2020) | 54.46 | 69.36 | 46.80 | 46.01 | 55.74 | 57.23 | 54.75 | 56.70 | 49.06 | - | - | - | - | - | - | -
AA-RMVSNet Wei et al. (2021) | 61.51 | 77.77 | 59.53 | 51.53 | 64.02 | 64.05 | 59.47 | 60.85 | 54.90 | 33.53 | 20.96 | 40.15 | 32.05 | 46.01 | 29.28 | 32.71
PatchmatchNet Wang et al. (2021b) | 53.15 | 66.99 | 52.64 | 43.24 | 54.87 | 52.87 | 49.54 | 54.21 | 50.81 | 32.31 | 23.69 | 37.73 | 30.04 | 41.80 | 28.31 | 32.29
UniMVSNet Peng et al. (2022) | 64.36 | 81.20 | 66.34 | 53.11 | 63.46 | 66.09 | 64.84 | 62.23 | 57.53 | 38.96 | 28.33 | 44.36 | 39.74 | 52.89 | 33.80 | 34.63
MVSTR Zhu et al. (2021) | 56.93 | 76.92 | 59.82 | 50.16 | 56.73 | 56.53 | 51.22 | 56.58 | 47.48 | 32.85 | 22.83 | 39.04 | 33.87 | 45.46 | 27.95 | 27.97
TransMVS Ding et al. (2022) | 63.52 | 80.92 | 65.83 | 56.94 | 62.54 | 63.06 | 60.00 | 60.20 | 58.67 | 37.00 | 24.84 | 44.59 | 34.77 | 46.49 | 34.69 | 36.62
MVSTER Wang et al. (2022) | - | - | - | - | - | - | - | - | - | 37.53 | 26.68 | 42.14 | 35.65 | 49.37 | 32.16 | 39.19
CostFormer(PatchMatchNet) | 56.27(+3.12) | 72.46 | 52.59 | 54.27 | 55.83 | 56.80 | 50.88 | 55.05 | 52.32 | 34.07(+1.76) | 24.05 | 39.20 | 32.17 | 43.95 | 28.62 | 36.46
CostFormer(PatchMatchNet*) | 57.10(+3.95) | 74.22 | 56.27 | 54.41 | 56.65 | 54.46 | 51.45 | 57.65 | 51.70 | 34.31(+2.00) | 26.77 | 39.13 | 31.58 | 44.55 | 28.79 | 35.03
CostFormer(UniMVSNet-) | 64.40(+0.04) | 81.45 | 66.22 | 53.88 | 62.94 | 66.12 | 65.35 | 61.31 | 57.90 | 39.55(+0.59) | 28.61 | 45.63 | 40.21 | 52.81 | 34.40 | 35.62
CostFormer(UniMVSNet*) | 64.51(+0.15) | 81.31 | 65.51 | 55.57 | 63.46 | 66.24 | 65.39 | 61.27 | 57.30 | 39.43(+0.47) | 29.18 | 45.21 | 39.88 | 53.38 | 34.07 | 34.87

Table 1: Quantitative results of different methods on the Tanks $\&$ Temples
benchmark (higher is better). * is pretrained on DTU and fine-tuned on
BlendedMVS. - is not pretrained on DTU and trained from scratch on BlendedMVS

However, there is no connection across local windows with fixed partitions.
Therefore, regular and shifted window partitions are used alternately to
enable cross-window connections. So at the next level, the window partition
configuration is shifted along the height, width, and depth axes by
$(\frac{h_{s}}{2},\frac{w_{s}}{2},\frac{d_{s}}{2})$. Depth-aware self-
attention will be computed in these shifted windows(DAS-MSA); the whole
process of DASTL can be formulated as:

$\widehat{X}^{l+1}=\text{DAS-MSA1}(\text{LN}(\text{DAS-
MSA2}(\text{LN}(X^{l}))))+X^{l}$ (13)
$X^{l+1}=\text{MLP}(\text{LN}(\widehat{X}^{l+1}))+\widehat{X}^{l+1}$ (14)

DAS-MSA1 and DAS-MSA2 correspond to multi-head Attention1 and Attention2
within a shifted window, respectively. Assuming the number of stages is $n$,
there are $n$ RDACT blocks in CostFormer.

###  3.3 Residual Regression Transformer

After aggregation, the cost $\widetilde{\mathcal{C}}\in\mathbb{R}^{HXWXD}$
will be used for depth regression. To further explore the spatial relation
under some depth, a Transformer block is applied to $\widetilde{\mathcal{C}}$
before softmax. Inspired by the RDACT, the whole process of Residual
Regression Transformer(RRT) can be formulated as:

$\widetilde{\mathcal{C}}_{k}=\text{RST}_{k}(\text{RT}_{k}(\widetilde{\mathcal{C}}_{k-1})),k=1,2,...,L$
(15)
$\widetilde{\mathcal{C}}_{out}=\text{RER}(\widetilde{\mathcal{C}}_{L})+\widetilde{\mathcal{C}}_{0}$
(16)

where $\text{RT}_{k}$ is the k-th Regression Transformer layer with regular
windows, $\text{RST}_{k}$ is the k-th Regression Transformer layer with
shifted windows, RER is the re-embedding layer to recover the depth dimension
from $\widetilde{\mathcal{C}}_{L}$, and it can be a 2D convolution with $D$
output channels.

RRT also computes self-attention in a local window. Compared with RDACT, RRT
focuses more on spatial relations. Compared with regular Swin Liu et al.
(2021) Transformer block, RRT treats the depth as a channel, the number of
channels is actually $1$ and this channel is squeezed before the Transformer.
The embedding parameters are set to fit the cost aggregation of different
iterations. If the embedding dimension number equals $D$,
$\widetilde{\mathcal{C}}_{out}$ can be simply formulated as:

$\widetilde{\mathcal{C}}_{out}=\widetilde{\mathcal{C}}_{L}+\widetilde{\mathcal{C}}_{0}$
(17)

As a stage may iterate many times with different depth hypotheses, the number
of RRT blocks should be set the same as the number of iterations. The whole
RRT is shown in the yellow window in Figure 2.

##  4 Training

###  4.1 Loss function

Final loss combines with the losses of all iterations at all stages and the
loss from the final refinement module:

$Loss=\overset{s}{\underset{k=1}{\sum}}\overset{n}{\underset{i=1}{\sum}}L_{i}^{k}+L_{ref}$
(18)

where $L_{i}^{k}$ is the regression or unification loss of the $i$-th
iteration at $k$-th stage. $L_{ref}$ is the regression or unification loss
from refinement module. If refinement module does not exist, the $L_{ref}$
loss is set to zero.

###  4.2 Common training settings

CostFormer is implemented by Pytorch Paszke et al. (2019). For RDACT, we set
the depth number at stages 3, 2, 1 as 4, 2, 2; patch size at height, width and
depth axes as 4, 4, 1; window size at height, width and depth axes as 7, 7, 2.
If the backbone is set as PatchMatchNet, embedding dimension number at stages
3, 2, 1 are set as 8, 8, 4. For RRT, we set the depth number as 2 at all
stages, patch size as 1 at all axes; window size as 8 at all axes. If the
backbone is set as PatchMatchNet, embedding dimension number at iteration 2,
2, 1 at stages 3, 2, 1 as 32, 64, 16, 16, 8. All models are trained on Nvidia
GTX V100 GPUs. After depth estimation, we reconstruct point clouds similar to
MVSNet Yao et al. (2018).

##  5 Experiments

In this section, we introduce multiple MVS datasets and evaluate our method on
these datasets. The results will be further reported in detail.

###  5.1 DATASETS

The datasets used in the evaluation are DTU Aanæs et al. (2016), BlendedMVS
Yao et al. (2020), ETH3D Schöps et al. (2017), Tanks $\&$ Temples Knapitsch et
al. (2017), and YFCC-100M Thomee et al. (2016). The DTU dataset is an indoor
multi-view stereo dataset with 124 different scenes, there are 49 views under
seven different lighting conditions in one scene. Tanks $\&$ Temples is
collected in a more complex and realistic environment, and it’s divided into
the intermediate and advanced set. ETH3D benchmark consists of calibrated
high-resolution images of scenes with strong viewpoint variations. It is
divided into training and test datasets. While the training dataset contains
13 scenes, the test dataset contains 12 scenes. BlendedMVS dataset is a large-
scale synthetic dataset, consisting of 113 indoor and outdoor scenes and split
into 106 training scenes and 7 validation scenes.

Methods | Acc. (mm) | Comp. (mm) | Overall (mm)
---|---|---|---
Furu Furukawa and Ponce (2010) | 0.613 | 0.941 | 0.777
Tola Tola et al. (2012) | 0.342 | 1.190 | 0.766
Gipuma Galliani et al. (2015) | 0.283 | 0.873 | 0.578
Colmap Schönberger and Frahm (2016) | 0.400 | 0.644 | 0.532
SurfaceNet Ji et al. (2017) | 0.450 | 1.040 | 0.745
MVSNet Yao et al. (2018) | 0.396 | 0.527 | 0.462
R-MVSNet Yao et al. (2019) | 0.383 | 0.452 | 0.417
P-MVSNet Luo et al. (2019) | 0.406 | 0.434 | 0.420
Point-MVSNet Chen et al. (2019) | 0.342 | 0.411 | 0.376
Fast-MVSNet Yu and Gao (2020) | 0.336 | 0.403 | 0.370
CasMVSNet Gu et al. (2020) | 0.325 | 0.385 | 0.355
UCS-Net Cheng et al. (2020) | 0.338 | 0.349 | 0.344
CVP-MVSNet Yang et al. (2020b) | 0.296 | 0.406 | 0.351
PVA-MVSNet Yi et al. (2020) | 0.379 | 0.336 | 0.357
PatchMatchNet Wang et al. (2021b) | 0.427 | 0.277 | 0.352
AA-RMVSNet Wei et al. (2021) | 0.376 | 0.339 | 0.357
UniMVSNet Peng et al. (2022) | 0.352 | 0.278 | 0.315
CostFormer(Based on PatchMatchNet) | 0.424 | 0.262 | 0.343 (+0.0093)
CostFormer(Based on CasMVSNet) | 0.378 | 0.313 |  0.345 (+0.0097)
COstFormer(Based on UniMVSNet) | 0.301 | 0.322 |  0.312 (+0.0035)

Table 2: Quantitative results of different methods on DTU.

###  5.2 Main Settings and Results on DTU

For the evaluation on the DTU Aanæs et al. (2016) evaluation set, we only use
the DTU training set. During the training phase, we set the image resolution
to 640 × 512. We compare our method to recent learning-based MVS methods,
including CasMVSNet Gu et al. (2020) and PatchMatchNet Wang et al. (2021b)
which are also set as backbones of CostFormer. We follow the evaluation
metrics provided by the DTU dataset. The quantitative results on the DTU
evaluation set are summarized in Table 2, which indicates that the plug-and-
play CostFormer improves the cost aggregation. Partial visualization results
of Table 2 are shown in Figure 3.

Complexity Analysis: For the complexity analysis of CostFormer, we plug it
into PatchMatchNet Wang et al. (2021b) and first compare the memory
consumption and run-time with this backbone. For a fair comparison, a fixed
input size of 1152 × 864 is used to evaluate the computational cost on a
single GPU of NVIDIA Telsa V100. Memory consumption and run-time of
PatchMatchNet Wang et al. (2021b) are 2323MB and 0.169s. They are only
increased to 2693MB and 0.231s by the plug-in.

Based on the reports of PatchMatchNet Wang et al. (2021b), we then get the
comparison results of other state-of-the-art learning-based methods. Memory
consumption and run-time are reduced by 61.9$\%$ and 54.8$\%$ compared to
CasMVSNet Gu et al. (2020), by 48.8$\%$ and 50.7$\%$ compared to UCSNet Cheng
et al. (2020) and by 63.5$\%$ and 77.3$\%$ compared to CVP-MVSNet Yang et al.
(2020b). Combining the results(lower is better) are shown in Table 3 and
Figure 1, GPU memory and run-time of CostFormer are set as 100$\%$.

Method |  GPU Memory ($\%$) |  Run-time ($\%$) | Overall (mm)
---|---|---|---
CasMVSNet Gu et al. (2020) |  262.47$\%$ |  221.24$\%$ | 0.355
UCSNet Cheng et al. (2020) |  195.31$\%$ |  202.84$\%$ | 0.344
CVP-MVSNet Yang et al. (2020b) |  273.97$\%$ |  440.53$\%$ | 0.351
Ours |  100.00$\%$ |  100.00$\%$ | 0.343

Table 3:  Comparison with other SOTA learning-based MVS methods on DTU.
Relationship between overall performance, GPU memory and run-time.

Comparison with Transformers We also compare CostFormer with other
Transformers Zhu et al. (2021); Wang et al. (2022); Ding et al. (2021); Liao
et al. (2022) which are used in MVS methods and not plug-and-play. For a fair
comparison, only direct improvements(higer is better) and incremental cost of
run time(low is better) from pure Transformers under similar depth hypotheses
are summarized in Table 4.

Method | Trans Improvement (mm) | Delta Time (s) | Delta Time (%)
---|---|---|---
MVSTR Zhu et al. (2021) | +0.0140 | +0.359s | +78.21%
TransMVS Ding et al. (2021) | +0.0160 | +0.367s | +135.42%
WT-MVSNet(CT) Liao et al. (2022) | +0.0130 | +0.265s | -
MVSTER(CNN Fusion) Wang et al. (2022) | +0.0040 | +0.016s | +13.34%
CostFormer(CNN Fusion) | +0.0097 | +0.062s | +36.69%

Table 4: Quantitative improvement of performance and incremental cost of run
time of different Transformers on DTU evaluation set.

###  5.3 Main Settings and Results on Tanks $\&$ Temples

For the evaluation on Tanks $\&$ Temples Knapitsch et al. (2017), we use the
DTU Aanæs et al. (2016) dataset and the Blended MVS Yao et al. (2020) dataset.
We compare our method to those recent learning-based MVS methods, including
PatchMatchNet Wang et al. (2021b) and UniMVSNet Peng et al. (2022) which are
also set as backbones of CostFormer. The quantitative results on the Tanks
$\&$ Temples Knapitsch et al. (2017) set are summarized in Table 1, which
indicates the robustness of CostFormer. Partial visualization results of Table
1 are shown in Figure 4. We would like to clarify that UniMVSNet- in Table 1
only uses BlendedMVS for training which uses less data (no DTU) than the
UniMVSNet baseline.

###  5.4 Main Settings and Results on ETH3D

We use the PatchMatchNet Wang et al. (2021b) as backbone and adopt the trained
model used in the Tanks $\&$ Temples dataset Knapitsch et al. (2017) to
evaluate the ETH3D Schöps et al. (2017) dataset. As shown in Table 5, our
method outperforms others on both the training and particularly challenging
test datasets(higher is better).

Methods | Training | Testing
---|---|---
F1 score $\uparrow$ |  Time(s) $\downarrow$ |  F1 score $\uparrow$ |  Time(s) $\downarrow$
MVE Fuhrmann et al. (2014) | 20.47 | 13278.69 | 30.37 | 10550.67
Gipuma Galliani et al. (2015) | 36.38 | 587.77 | 45.18 | 689.75
PMVS Furukawa and Ponce (2010) | 46.06 | 836.66 | 44.16 | 957.08
COLMAP Schönberger and Frahm (2016) | 67.66 | 2690.62 | 73.01 | 1658.33
PVSNet Xu and Tao (2020) | 67.48 | - | 72.08 | 829.5
IterMVS Wang et al. (2021a) | 66.36 | - | 74.29 | -
PatchMatchNet Wang et al. (2021b) | 64.21 | 452.63 | 73.12 | 492.52
PatchMatch-RL Lee et al. (2021) | 67.78 | - | 72.38 | -
CostFormer(Ours) | 68.92(+4.71) | 566.18 | 75.24(+2.12) | 547.64

Table 5: Quantitative results of different methods on ETH3D.

###  5.5 Main Settings and Results on BlendedMVS dataset

We use the model used in ETH3D. On BlendedMVS Yao et al. (2020) evaluation
set, we set $N=5$ and image resolution as 576 × 768. End point error (EPE), 1
pixel error (e1), and 3 pexels error (e3) are used as the evaluation metrics.
Quantitative results(lower is better) of different methods are shown in Table
6.

Method | EPE | e1 (%) | e3 (%)
---|---|---|---
MVSNet Yao et al. (2018) | 1.49 | 21.98 | 8.32
MVSNet-s Darmon et al. (2021) | 1.35 | 25.91 | 8.55
CVP-MVSNet Yang et al. (2020a) | 1.90 | 19.73 | 10.24
VisMVSNet Zhang et al. (2020) | 1.47 | 18.47 | 7.59
CasMVSNet Gu et al. (2020) | 1.98 | 15.25 | 7.60
EPPMVSNet Ma et al. (2021) | 1.17 | 12.66 | 6.20
TransMVSNet Ding et al. (2021) | 0.73 | 8.32 | 3.62
CostFormer(Based on PatchmatchNet) | 0.84 | 12.37 | 4.59
CostFormer(Based on UniMVSNet) | 0.43 | 7.05 | 2.70

Table 6: Quantitative results of different methods on BlendedMVS

##  6 Conclusion

In this work, we explore whether cost Transformer can improve the cost
aggregation and propose a novel CostFormer with the cascade RDACT and RRT
modules. The experimental results on DTU Aanæs et al. (2016) , Tanks $\&$
Temples Knapitsch et al. (2017), ETH3D Schöps et al. (2017), and BlendedMVS
Yao et al. (2020) show that our method is competitive, efficient, and plug-
and-play. Cost Transformer can be your need for better cost aggregation in
multi-view stereo.

## References

  * Aanæs et al. [2016] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl.  Large-scale data for multiple-view stereopsis.  Int. J. Comput. Vis., 120(2):153–168, 2016.
  * Chen et al. [2019] Rui Chen, Songfang Han, Jing Xu, and Hao Su.  Point-based multi-view stereo network.  In ICCV, pages 1538–1547. IEEE, 2019.
  * Cheng et al. [2020] Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su.  Deep stereo using adaptive thin volume representation with uncertainty awareness.  In CVPR, pages 2521–2531. IEEE, 2020.
  * Cho et al. [2021] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim.  Cats: Cost aggregation transformers for visual correspondence.  Advances in Neural Information Processing Systems, 34, 2021.
  * Darmon et al. [2021] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry.  Deep multi-view stereo gone wild.  CoRR, abs/2104.15119, 2021.
  * Ding et al. [2021] Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and Xiao Liu.  Transmvsnet: Global context-aware multi-view stereo network with transformers.  arXiv preprint arXiv:2111.14600, 2021.
  * Ding et al. [2022] Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and Xiao Liu.  Transmvsnet: Global context-aware multi-view stereo network with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8585–8594, 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * Fuhrmann et al. [2014] Simon Fuhrmann, Fabian Langguth, and Michael Goesele.  Mve - a multi-view reconstruction environment.  In GCH, pages 11–18. Eurographics Association, 2014.
  * Furukawa and Ponce [2010] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multiview stereopsis.  IEEE Trans. Pattern Anal. Mach. Intell., 32(8):1362–1376, 2010\.
  * Galliani et al. [2015] Silvano Galliani, Katrin Lasinger, and Konrad Schindler.  Massively parallel multiview stereopsis by surface normal diffusion.  In ICCV, pages 873–881. IEEE Computer Society, 2015.
  * Gu et al. [2020] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495–2504, 2020.
  * Hosni et al. [2012] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz.  Fast cost-volume filtering for visual correspondence and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2):504–511, 2012.
  * Ji et al. [2017] Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang.  Surfacenet: An end-to-end 3d neural network for multiview stereopsis.  In ICCV, pages 2326–2334. IEEE Computer Society, 2017.
  * Knapitsch et al. [2017] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: benchmarking large-scale scene reconstruction.  ACM Trans. Graph., 36(4):78:1–78:13, 2017.
  * Lee et al. [2021] Jae Yong Lee, Joseph DeGol, Chuhang Zou, and Derek Hoiem.  Patchmatch-rl: Deep mvs with pixelwise depth, normal, and visibility.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2021.
  * Li et al. [2021] Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell H Taylor, and Mathias Unberath.  Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6197–6206, 2021.
  * Liao et al. [2022] Jinli Liao, Yikang Ding, Yoli Shavit, Dihe Huang, Shihao Ren, Jia Guo, Wensen Feng, and Kai Zhang.  Wt-mvsnet: Window-based transformers for multi-view stereo, 2022.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * Luo et al. [2019] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo.  P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo.  In ICCV, pages 10451–10460. IEEE, 2019.
  * Ma et al. [2021] Xinjun Ma, Yue Gong, Qirui Wang, Jingwei Huang, Lei Chen, and Fan Yu.  Epp-mvsnet: Epipolar-assembling based depth prediction for multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5732–5740, 2021.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * Peng et al. [2022] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang.  Rethinking depth estimation for multi-view stereo: A unified representation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * Scharstein and Szeliski [2002] Daniel Scharstein and Richard Szeliski.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision, 47(1):7–42, 2002.
  * Schönberger and Frahm [2016] Johannes L. Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, pages 4104–4113. IEEE Computer Society, 2016.
  * Schöps et al. [2017] Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger.  A multi-view stereo benchmark with high-resolution images and multi-camera videos.  In CVPR, pages 2538–2547. IEEE Computer Society, 2017.
  * Sun et al. [2021] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021.
  * Thomee et al. [2016] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: the new data in multimedia research.  Commun. ACM, 59(2):64–73, 2016.
  * Tola et al. [2012] Engin Tola, Christoph Strecha, and Pascal Fua.  Efficient large-scale multi-view stereo for ultra high-resolution image sets.  Mach. Vis. Appl., 23(5):903–920, 2012.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wang et al. [2021a] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, and Marc Pollefeys.  Itermvs: Iterative probability estimation for efficient multi-view stereo.  arXiv preprint arXiv:2112.05126, 2021.
  * Wang et al. [2021b] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys.  Patchmatchnet: Learned multi-view patchmatch stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14194–14203, 2021.
  * Wang et al. [2022] Xiaofeng Wang, Zheng Zhu, Fangbo Qin, Yun Ye, Guan Huang, Xu Chi, Yijia He, and Xingang Wang.  Mvster: Epipolar transformer for efficient multi-view stereo, 2022.
  * Wei et al. [2021] Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang.  Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6187–6196, 2021.
  * Xu and Tao [2020] Qingshan Xu and Wenbing Tao.  Pvsnet: Pixelwise visibility-aware multi-view stereo network.  CoRR, abs/2007.07714, 2020.
  * Yan et al. [2020] Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu-Wing Tai.  Dense hybrid recurrent multi-view stereo net with dynamic consistency checking.  In European Conference on Computer Vision, pages 674–689. Springer, 2020.
  * Yang et al. [2020a] Jiayu Yang, Wei Mao, Jose M Alvarez, and Miaomiao Liu.  Cost volume pyramid based depth inference for multi-view stereo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4877–4886, 2020.
  * Yang et al. [2020b] Jiayu Yang, Wei Mao, Jose M. Alvarez, and Miaomiao Liu.  Cost volume pyramid based depth inference for multi-view stereo.  In CVPR, pages 4876–4885. IEEE, 2020.
  * Yao et al. [2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018.
  * Yao et al. [2019] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In CVPR, pages 5525–5534. Computer Vision Foundation / IEEE, 2019\.
  * Yao et al. [2020] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  Computer Vision and Pattern Recognition (CVPR), 2020.
  * Yi et al. [2020] Hongwei Yi, Zizhuang Wei, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu-Wing Tai.  Pyramid multi-view stereo net with self-adaptive view aggregation.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV (9), volume 12354 of Lecture Notes in Computer Science, pages 766–782. Springer, 2020.
  * Yu and Gao [2020] Zehao Yu and Shenghua Gao.  Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement.  In CVPR, pages 1946–1955. IEEE, 2020.
  * Zhang et al. [2020] Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang.  Visibility-aware multi-view stereo network.  British Machine Vision Conference (BMVC), 2020.
  * Zhu et al. [2021] Jie Zhu, Bo Peng, Wanqing Li, Haifeng Shen, Zhe Zhang, and Jianjun Lei.  Multi-view stereo with transformer.  ArXiv, abs/2112.00336, 2021.


ATL-PHYS-PROC-2021-016

Treatment of top-quark backgrounds in extreme phase spaces:
the “top $p_{\mathrm{T}}$ reweighting” and novel data-driven estimations in
ATLAS and CMS

Leonid Serkin
on behalf of the ATLAS and CMS Collaborations111Copyright [2021] CERN for the
benefit of the ATLAS and CMS Collaborations. Reproduction of this article or
parts of it is allowed as specified in the CC-BY-4.0 license

INFN Gruppo Collegato di Udine, Sezione di Trieste, Udine and ICTP, Trieste
Strada Costiera 11, Trieste 34151, Italy

> The top quark plays an important role in searches for physics beyond the SM,
> both as a dominant background and as a key signature for the signal. The
> most notable feature found in the top physics analyses in both ATLAS and CMS
> Collaborations: the disagreement between simulation and data of the top
> quark $p_{\mathrm{T}}$ spectrum - is highlighted. A reweighting procedure
> which significantly improves the agreement between simulation and data, also
> known as the “top $p_{\mathrm{T}}$ reweighting”, is summarised. Commonly
> raised points concerning the reweighting to fixed-order predictions are
> discussed, and several refined approaches are mentioned. An overview of
> several data-driven methods developed and used to estimate the
> $t\overline{t}$ background in regions with large jet and $b$-jet
> multiplicities and/or high top quark $p_{\mathrm{T}}$ is presented.

> PRESENTED AT
>
>
>
>
> $13^{\mathrm{th}}$ International Workshop on Top Quark Physics
> IPPP Durham (online version), UK, September 14–18, 2020

##  1 Introduction

With a mass close to the scale of electroweak symmetry breaking, the top
quark, besides having a large coupling to the Standard Model (SM) Higgs boson
[1, 2], is predicted to have large couplings to new particles hypothesized in
many models beyond the Standard Model (BSM). Possible new phenomena may
enhance the cross sections over SM predictions for various processes involving
or decaying into top quarks, as detailed in Ref. [3] and references therein.
Hence, the top quark plays an important role in searches for BSM physics, both
as dominant background and as key signature for signal.

In the following, the reweighting procedure which significantly improves the
agreement between simulation and data, also known as the “top $p_{\mathrm{T}}$
reweighting”, is discussed. Recent results by the ATLAS [4] and CMS [5]
experiments at the LHC using $\sqrt{s}=13$ TeV data that apply the top
$p_{\mathrm{T}}$ reweighting are summarised. Several issues concerning the
reweighting to fixed-order predictions are mentioned and future approaches are
discussed. Secondly, novel data-driven techniques for the estimation of the
$t\overline{t}$ production background are presented. The focus is made on
analyses using $\sqrt{s}=13$ TeV data and exploring high $p_{\mathrm{T}}$
and/or large multiplicity regimes.

##  2 Top quark $p_{\mathrm{T}}$ reweighting

One of the most notable features found in the top physics analyses is the
disagreement between simulation and data of the top quark pair production. In
particular, the $p_{\mathrm{T}}$ distribution of the top quark is softer than
predicted by MC event generators using NLO-accurate matrix elements with
parton showering (denoted as NLO MC+PS), a trend observed in both resolved and
boosted regimes and consistent among ATLAS and CMS . This mismodelling, first
observed around 2012 in differential $t\overline{t}$ cross-section
measurements (see Ref. [6] and references therein), largely affects the
data/MC agreement, as seen in Fig. 1(a), and hence is one of the main
uncertainties limiting precise measurements.

\begin{overpic}[height=140.92792pt]{HT.png} \put(60.0,-7.0){(a)} \end{overpic}

\begin{overpic}[height=140.92792pt]{cms_pt.png} \put(80.0,-7.0){(b)}
\end{overpic}

Figure 1:  (a) A clear slope due to the top quark $p_{\mathrm{T}}$
mismodelling in the distribution of the scalar sum of jet $p_{\mathrm{T}}$
[7]. (b) Ratio between 13 TeV data and Powheg simulation for the differential
top quark pair cross section as a function of $p_{\mathrm{T}}$ of the top
quark [8].

A short-term solution for this mismodelling was proposed in Refs. [12, 13]: an
empirical reweighting for top quark pairs based on the $p_{\mathrm{T}}$
spectrum of generated-level top quarks, the so-called “top $p_{\mathrm{T}}$
reweighting”. Initially, the reweighting was applied to simulated
$t\overline{t}$ events based on the ratio of measured differential cross
sections [14, 15, 16] between data and the simulation as a function of top
quark $p_{\mathrm{T}}$, as seen in Fig. 1(b). With the advent of the
differential cross sections computed at next-to-NLO accuracy in pQCD (denoted
as NNLO QCD) [17], and those including electroweak (EW) corrections computed
at NLO accuracy (denoted as NNLO QCD + NLO EW) [18], a reweighting of the
$t\overline{t}$ samples in order to match their top quark $p_{\mathrm{T}}$
distributions to that predicted by the best available fixed-order calculation
became available.

Besides significantly improving the agreement between simulation and data, the
top $p_{\mathrm{T}}$ reweighting is also able to reduce the size of the
modelling systematics affecting the simulation, improving the sensitivity as
well as the reliability of the systematics model. The treatment of the top
$p_{\mathrm{T}}$ reweighting in both ATLAS and CMS analyses changes depending
on the use case, i.e. correcting efficiencies or acceptance in $t\overline{t}$
measurements or searches. As an example, the reweighting of parton-level
kinematics to differential cross-section measurements is not recommended for
searches in high $p_{\mathrm{T}}$ regimes due to the lack of statistics in
deriving the weights in the tails. If no control region in data can be
identified during a search, the analysis in CMS would correct the nominal
sample to the best available fixed-order prediction by default, while ATLAS
would consider it as systematic uncertainty. A summary of recent ATLAS and CMS
searches and measurements applying different top $p_{\mathrm{T}}$ reweightings
and their associated uncertainties are presented in Table 1.

Table 1: A non-comprehensive list of recent ATLAS and CMS searches and measurements applying different top $p_{\mathrm{T}}$ reweightings and their associated uncertainties.  Analysis / search | Top $p_{\mathrm{T}}$ reweighting | Uncertainty
---|---|---
SM $t\overline{t}t\overline{t}$
1 and 2 OS leptons
CMS Collaboration [9] |  Function derived from differential measurements |  Allowing the correction function to vary within a $\pm 1\sigma$ uncertainty
BSM $t\overline{t}t\overline{t}$
0 and 1 lepton final state
ATLAS Collaboration [10] |  NNLO QCD |  Full difference between applying and not applying
SM $t\overline{t}H$
multi-lepton final state
CMS Collaboration [11] |  NNLO QCD + NLO EW |  Difference between nominal and reweighted sample
$t\overline{t}$ cross-section
1 lepton final state
ATLAS Collaboration [7] |  NNLO QCD + NLO EW |  Symmetrised full difference between nominal and the reweighted sample

Usually both ATLAS and CMS assign a systematic uncertainty derived from the
difference between the applying and not the top $p_{\mathrm{T}}$ reweighing;
if used in a maximum likelihood fit, this nuisance is profiled and is commonly
accepted that the nuisance parameter responsible for this corrections could be
pulled indicating the need for this correction in data (see Fig. 2).

\begin{overpic}[height=140.92792pt]{nuis_atlas.png} \put(90.0,-9.0){(a)}
\end{overpic}

\begin{overpic}[height=140.92792pt]{nuis_cms.png} \put(110.0,-9.0){(b)}
\end{overpic}

Figure 2:  (a) Ranking plot showing the effect of the ten most important
systematic uncertainties in the 13 TeV $t\overline{t}$ cross-section
measurement by ATLAS [7]. The top $p_{\mathrm{T}}$ reweighting parameter (5th
from the top) is pulled towards the fixed-order prediction, as expected. (b)
Normalised pulls and constraints of the nuisance parameters for the 13 TeV
$t\overline{t}$ cross-section measurement by CMS [19]. The nuisance parameter
for the $p_{\mathrm{T}}$ distribution of the top quarks (5th from the left) is
pulled by one standard deviation, as expected.

###  2.1 Considerations on reweighting to fixed-order predictions

The commonly raised points concerning the reweighting to fixed-order
predictions are summarised in the following:

  * •

The latest fixed-order calculations recommend different functional forms for
the renormalisation and factorisation scales for different observables; the
parton distribution functions (PDF), the top mass settings as well as the
scale variations and scale choices are not always easily available in the
fixed-order predictions, nor match the ATLAS and CMS settings for the MC
generation;

  * •

In the current way the top $p_{\mathrm{T}}$ reweighting is applied, only the
top $p_{\mathrm{T}}$ differential distribution is corrected to the given
fixed-order calculation: yet, the question about other (partially correlated)
variables (e.g. $t\overline{t}$-system mass) remains open (see Fig. 3). After
the top $p_{\mathrm{T}}$ reweighting is applied, analysers within both
Collaborations are requested to monitor the change in agreement between data
and the corrected MC in other distributions and make sure that the reweighting
does not spoil the initial agreement.

\begin{overpic}[height=140.92792pt]{pt_example.png} \put(90.0,-7.0){(a)}
\end{overpic}

\begin{overpic}[height=140.92792pt]{mass_example.png} \put(90.0,-7.0){(b)}
\end{overpic}

Figure 3:  (a) Distribution of the top quark $p_{\mathrm{T}}$. The bottom
panel shows the ratio of an example NLO MC+PS simulation with respect to the
NNLO QCD prediction without (red) and with (black) NLO EW corrections. (b)
Distribution of the $t\overline{t}$-system mass. The bottom panel shows the
ratio of the NNLO QCD + NLO EW prediction with respect to an example NLO MC+PS
simulation with (red) and without (blue) applying the top $p_{\mathrm{T}}$
reweighting, derived as the black line in Fig. 3(a). Both distributions are
normalised to unity.

Several refined approaches to the reweighting are possible, and a list of
options to be explored is presented below:

  * •

Given that the baseline Powheg v2 [20] NLO MC simulation of the inclusive
$t\overline{t}$ production in both ATLAS and CMS uses the same factorisation
and renormalisation scales ($\sqrt{m_{top}^{2}+p_{T,top}^{2}}$), having fixed-
order calculations with the same scale for all variables would be helpful; the
same choice of PDFs and top mass would help the Collaborations apply the
reweighting in a consistent manner (e.g. ATLAS and CMS use a top mass of 172.5
GeV in Powheg v2, while the fixed-order calculations use the value of 173.3
GeV as an input parameter);

  * •

Given that dedicated fixed-order calculations for the $t\overline{t}$-system
mass distribution are available [18], a two-dimensional reweighting (i.e.
based on top $p_{\mathrm{T}}$ and $t\overline{t}$-system mass) could account
for correlations among these variables; an interative, recursive reweighting
to these two distributions could be tested as well, giving a MC prediction
which matches both top $p_{\mathrm{T}}$ and $t\overline{t}$-system mass fixed-
order predictions.

##  3 Data-driven estimations of the $t\overline{t}$ background

The MC simulation-based approach at NLO accuracy in QCD for the prediction of
the inclusive $t\overline{t}$ background is not expected to model well the
very high jet and $b$-jet multiplicity regions. Given the current lack of
multi-leg calculations, the MC simulation-based approach relies on the
description of such large multiplicities through the PS formalism with
consequently large uncertainties. Therefore, several data-driven methods have
been developed and used to estimate the $t\overline{t}$ background in regions
with large jet and $b$-jet multiplicities and/or high top quark
$p_{\mathrm{T}}$. Recent examples of novel estimations of the $t\overline{t}$
background are presented in Table 2. The different data-driven techniques
perform well: data agree with the SM expectation within the uncertainties,
validating the overall data-driven procedures and the assumptions made, and
allowing to search for new physics in extreme phase spaces probed in each of
the searches.

Table 2: A non-comprehensive list of recent ATLAS and CMS searches and measurements applying a data-driven estimation of the $t\overline{t}$ background.  Analysis / search | Data-driven technique | Uncertainty
---|---|---
$Z^{\prime}\rightarrow t\overline{t}$
0 leptons final state
ATLAS Collaboration [21] |  Background spectrum derived from data by fitting a smoothly falling function to the mass spectrum of the $t\overline{t}$ system distribution |  Choice of functional form and the fit range; spurious studies by performing signal-plus-background (S+B) fits constructed under a B-only hypothesis
$X\rightarrow HH\rightarrow b\overline{b}W^{+}W^{-}$
1 lepton final state
CMS Collaboration [22] |  $t\overline{t}$ background modelled with two-dimensional templates as conditional probabilities using kernel estimation; yields extracted from data using a maximum likelihood fit |  Morphing the background templates, derived by repeating their construction for different assumptions on jet $p_{\mathrm{T}}$ spectrum and resonance mass
SM and BSM $t\overline{t}t\overline{t}$
1 and 2 OS leptons
ATLAS Collaboration [23] |  Assumption that the probability of $b$-tagging a jet (measured as a function of jet and event properties) in $t\overline{t}$+jets event is essentially independent of the number of additional jets |  A full set of systematic uncertainties is derived for the estimate by repeating the procedure on MC simulated events with systematic variations applied
SUSY $t\overline{t}t\overline{t}$
1 lepton final state
CMS Collaboration [24] |  Modified version of an “ABCD” method using two uncorrelated variables; any residual correlation taken into account by correction factors |  Tests performed using data control samples in regions that are kinematically similar but have only a very small potential contribution from signal events.

###  3.1 Summary

Since 2012, the mismodelling of the top quark $p_{\mathrm{T}}$ distribution in
data is still one of the main features observed in data. Both ATLAS and CMS
analyses correct the top quark $p_{\mathrm{T}}$ in MC by using the reweighting
to fixed-order predictions or to differential measurements, and refined
approaches to the reweightings are currently possible. The discrepancy seems
to be originated due to missing QCD and EW corrections in the MC generators,
and tuning efforts may improve the agreement with data.

In order to search for new physics in extreme phase spaces without dealing
with the so-called “top $p_{\mathrm{T}}$ reweighting”, several data-driven
estimation techniques were developed within both ATLAS and CMS Collaborations.
These efforts tackle the modelling of the $t\overline{t}$ background in a new
way, and their development should be continued and widely tested.

## References

  * [1] CMS Collaboration, Phys. Rev. Lett. 120 (2018) 231801.
  * [2] ATLAS Collaboration, Phys. Lett. B 784 (2018) 173.
  * [3] L. Serkin, arXiv:1901.01765 [hep-ex] (2019).
  * [4] ATLAS Collaboration, JINST 3 S08003, (2008).
  * [5] CMS Collaboration, JINST 3 S08004, (2008).
  * [6] Proceedings of TOP 2013, DESY-PROC-2014-02 (2014).
  * [7] ATLAS Collaboration, Phys. Lett. B 810 (2020) 135797.
  * [8] CMS Collaboration, CMS Top Quark Physics Summary Figures (as of 10.2020).
  * [9] CMS Collaboration, JHEP 11 (2019) 082.
  * [10] ATLAS Collaboration, JHEP 07 (2018) 089.
  * [11] CMS Collaboration, arXiv:2011.03652 [hep-ex] (2020).
  * [12] CMS Collaboration, CMS-PAS-TOP-12-030 (2012).
  * [13] ATLAS Collaboration, ATLAS-CONF-2014-011 (2014).
  * [14] ATLAS Collaboration, Phys. Rev. D 90 (2014) 072004.
  * [15] CMS Collaboration, Phys. Rev. D 95, 092001 (2017).
  * [16] CMS Collaboration, JHEP 04 (2018) 060.
  * [17] M. Czakon et al., JHEP 04 (2017) 071.
  * [18] M. Czakon et al., JHEP 10 (2017) 186.
  * [19] CMS Collaboration, Eur. Phys. J. C 79 (2019) 368.
  * [20] S. Alioli et al., JHEP 06 (2010) 043.
  * [21] ATLAS Collaboration, JHEP 10 (2020) 061.
  * [22] CMS Collaboration, JHEP 10 (2019) 125.
  * [23] ATLAS Collaboration, Phys. Rev. D 99 (2019) 052009.
  * [24] CMS Collaboration, Phys. Rev. D 101 (2020) 052010.