# Real-Time High-Resolution Background Matting

Shanchuan Lin Andrey Ryabtsev* Soumyadip Sengupta
Brian Curless Steve Seitz Ira Kemelmacher-Shlizerman
University of Washington
<EMAIL_ADDRESS>Equal
contribution.

###### Abstract

We introduce a real-time, high-resolution background replacement technique
which operates at 30fps in 4K resolution, and 60fps for HD on a modern GPU.
Our technique is based on background matting, where an additional frame of the
background is captured and used in recovering the alpha matte and the
foreground layer. The main challenge is to compute a high-quality alpha matte,
preserving strand-level hair details, while processing high-resolution images
in real-time. To achieve this goal, we employ two neural networks; a base
network computes a low-resolution result which is refined by a second network
operating at high-resolution on selective patches. We introduce two large-
scale video and image matting datasets: VideoMatte240K and PhotoMatte13K/85.
Our approach yields higher quality results compared to the previous state-of-
the-art in background matting, while simultaneously yielding a dramatic boost
in both speed and resolution. Our code and data is available at
https://grail.cs.washington.edu/projects/background-matting-v2/

##  1 Introduction

Background replacement, a mainstay in movie special effects, now enjoys wide-
spread use in video conferencing tools like Zoom, Google Meet, and Microsoft
Teams. In addition to adding entertainment value, background replacement can
enhance privacy, particularly in situations where a user may not want to share
details of their location and environment to others on the call. A key
challenge of this video conferencing application is that users do not
typically have access to a green screen or other physical props used to
facilitate background replacement in movie special effects.

While many tools now provide background replacement functionality, they yield
artifacts at boundaries, particularly in areas where there is fine detail like
hair or glasses (Figure 1). In contrast, traditional image matting methods [6,
16, 17, 30, 9, 2, 7] provide much higher quality results, but do not run in
real-time, at high resolution, and frequently require manual input. In this
paper, we introduce the first fully-automated, real-time, high-resolution
matting technique, producing state-of-the-art results at 4K (3840$\times$2160)
at 30fps and HD (1920$\times$1080) at 60fps. Our method relies on capturing an
extra background image to compute the alpha matte and the foreground layer, an
approach known as background matting.

Designing a neural network that can achieve real-time matting on high-
resolution videos of people is extremely challenging, especially when fine-
grained details like strands of hair are important; in contrast, the previous
state-of-the-art method [28] is limited to 512$\times$512 at 8fps. Training a
deep network on such a large resolution is extremely slow and memory
intensive. It also requires large volumes of images with high-quality alpha
mattes to generalize; the publicly available datasets [33, 25] are too
limited.

Since it is difficult to collect a high-quality dataset with manually curated
alpha mattes in large quantities, we propose to train our network with a
series of datasets, each with different characteristics. To this end, we
introduce VideoMatte240K and PhotoMatte13K/85 with high-resolution alpha
mattes and foreground layers extracted with chroma-key software. We first
train our network on these larger databases of alpha mattes with significant
diversity in human poses to learn robust priors. We then train on publicly
available datasets [33, 25] that are manually curated to learn fine-grained
details.

To design a network that can handle high-resolution images in real-time, we
observe that relatively few regions in the image require fine-grained
refinement. Therefore, we introduce a base network that predicts the alpha
matte and foreground layer at lower resolution along with an error prediction
map which specifies areas that may need high-resolution refinement. A
refinement network then takes the low-resolution result and the original image
to generate high-resolution output only at select regions.

We produce state-of-the-art background matting results in real-time on
challenging real-world videos and images of people. We will release our
VideoMatte240K and PhotoMatte85 datasets and our model implementation.

##  2 Related Work

Background replacement can be achieved with segmentation or matting. While
binary segmentation is fast and efficient, the resulting composites have
objectionable artifacts. Alpha matting can produce visually pleasing
composites but often requires either manual annotations or a known background
image. In this section, we discuss related works that perform background
replacement with segmentation or matting.

Segmentation. The literature in both instance and semantic segmentation is
vast and out of scope for this paper, so we will review the most relevant
works. Mask RCNN [11] is still a top choice for instance segmentation while
DeepLabV3+ [5] is a state-of-the-art semantic segmentation network. We
incorporate the Atrous Spatial Pyramid Pooling (ASPP) module from DeepLabV3
[4] and DeepLabV3+ within our network. Since segmentation algorithms tend to
produce coarse boundaries especially at higher resolutions, Kirillov et al.
presented PointRend [15] which samples points near the boundary and
iteratively refines the segmentation. This produces high-quality segmentation
for large image resolutions with significantly cheaper memory and computation.
Our method adopts this idea to the matting domain via learned refinement-
region selection and a convolutional refinement architecture that improves the
receptive field. Specific applications of human segmentation and parsing have
also received considerable attention in recent works [34, 19].

Trimap-based matting. Traditional (non-learning based) matting algorithms [6,
16, 17, 30, 9, 2, 7] require manual annotation (a trimap) and solve for the
alpha matte in the ‘unknown’ region of the trimap. Different matting
techniques are reviewed in the survey by Wang and Cohen [32]. Xu et al. [33]
introduced a matting dataset and used a deep network with a trimap input to
predict the alpha matte. Many recent approaches rely on this dataset to learn
matting, e.g., Context-Aware Matting [13], Index Matting [21], sampling-based
matting [31] and opacity propagation-based matting [18]. Although the
performance of these methods depends on the quality of the annotations, some
recent methods consider coarse [20] or faulty human annotations [3] to predict
the alpha matte.

Matting without any external input. Recent approaches have also focused on
matting humans without any external input. Portrait matting without a trimap
[36, 29] is one of the more successful applications due to less variability
among portrait images compared to full body humans. Soft segmentation for
natural images had also been explored in [1]. Recent approaches like Late
Fusion Matting [35] and HAttMatting [25] aim to solve for the alpha matte
directly from the image, but these approaches can often fail to generalize as
shown in [28].

Matting with a known natural background. Matting with known natural background
had been previously explored in [24], Bayesian matting [7] and Poisson matting
[30, 10] which also requires a trimap. Recently Sengupta et al. [28]
introduced Background Matting (BGM) where an additional background image is
captured and it provides a significant cue to predict the alpha matte and the
foreground layer. Although this method showed high-quality matting results,
the architecture is limited to 512$\times$512 resolution and runs only at
8fps. In contrast, we introduce a real-time unified matting architecture that
operates on 4K videos at 30fps and HD videos at 60fps, and produces higher
quality results than BGM.

##  3 Our Dataset

(a) VideoMatte240K (b) PhotoMatte13K/85

Figure 1: We introduce two large-scale matting datasets containing 240k unique
frames and 13k unique photos.

Since it is extremely difficult to obtain a large-scale, high-resolution,
high-quality matting dataset where the alpha mattes are cleaned by human
artists, we rely on multiple datasets including our own collections and
publicly available datasets.

Publicly available datasets. The Adobe Image Matting (AIM) dataset [33]
provides 269 human training samples and 11 test samples, averaging around
1000$\times$1000 resolution. We also use a humans-only subset of
Distinctions-646 [25] consisting of 362 training and 11 test samples,
averaging around 1700$\times$2000 resolution. The mattes were created manually
and are thus high-quality. However 631 training images are not enough to learn
large variations in human poses and finer details at high resolution, so we
introduce 2 additional datasets.

VideoMatte240K. We collect 484 high-resolution green screen videos and
generate a total of 240,709 unique frames of alpha mattes and foregrounds with
chroma-key software Adobe After Effects. The videos are purchased as stock
footage or found as royalty-free materials online. 384 videos are at 4K
resolution and 100 are in HD. We split the videos by 479 : 5 to form the train
and validation sets. The dataset consists of a vast amount of human subjects,
clothing, and poses that are helpful for training robust models. We are
releasing the extracted alpha mattes and foregrounds as a dataset to the
public. To our knowledge, our dataset is larger than all existing matting
datasets publicly available by far, and it is the first public video matting
dataset that contains continuous sequences of frames instead of still images,
which can be used in future research to develop models that incorporate motion
information.

PhotoMatte13K/85. We acquired a collection of 13,665 images shot with studio-
quality lighting and cameras in front of a green-screen, along with mattes
extracted via chroma-key algorithms with manual tuning and error repair. We
split the images by 13,165 : 500 to form the train and validation sets. These
mattes contain a narrow range of poses but are high resolution, averaging
around 2000$\times$2500, and include details such as individual strands of
hair. We refer to this dataset as PhotoMatte13K. However privacy and licensing
issues prevent us from sharing this set; thus, we also collected an additional
set of 85 mattes of similar quality for use as a test set, which we are
releasing to the public as PhotoMatte85. In Figure 1 we show examples from the
VideoMatte240K and PhotoMatte13K/85 datasets.

We crawl 8861 high-resolution background images from Flickr and Google and
split them by 8636 : 200 : 25 to use when constructing the train, validation,
and test sets. We will release the test set in which all images have a CC
license (see appendix for details).

##  4 Our Approach

Figure 2: The base network $G_{\text{base}}$ (blue) operates on the
downsampled input to produce coarse-grained results and an error prediction
map. The refinement network $G_{\text{refine}}$ (green) selects error-prone
patches and refines them to the full resolution.

(a) Coarse |  (b) Selection |  (c) Refined
---|---|---

Figure 3: We only refine on error-prone regions (b) and directly upsample the
rest to save computation.

Given an image $I$ and the captured background $B$ we predict the alpha matte
$\alpha$ and the foreground $F$, which will allow us to composite over any new
background by $I^{\prime}=\alpha F+(1-\alpha)B^{\prime}$, where $B^{\prime}$
is the new background. Instead of solving for the foreground directly, we
solve for foreground residual $F^{R}=F-I$. Then, $F$ can be recovered by
adding $F^{R}$ to the input image $I$ with suitable clamping:
$F=\max(\min(F^{R}+I,1),0)$. We find this formulation improves learning, and
allows us to apply a low-resolution foreground residual onto a high-resolution
input image through upsampling, aiding our architecture as described later.

Matting at high resolution is challenging, as applying a deep network directly
incurs impractical computation and memory consumption. As Figure 3 shows,
human mattes are usually very sparse, where large areas of pixels belong to
either background ($\alpha=0$) or foreground ($\alpha=1$), and only a few
areas involve finer details, e.g., around the hair, glasses, and person’s
outline. Thus instead of designing one network that operates on high-
resolution images, we introduce two networks; one operates at lower-resolution
and another only operates on selected patches at the original resolution based
on the prediction of the previous network.

The architecture consists of a base network $G_{\text{base}}$ and a refinement
network $G_{\text{refine}}$. Given the original image $I$ and the captured
background $B$, we first downsample by a factor of $c$ to $I_{c}$ and $B_{c}$.
The base network $G_{\text{base}}$ takes $I_{c}$ and $B_{c}$ as input and
predicts coarse-grained alpha matte $\alpha_{c}$, foreground residual
$F^{R}_{c}$, an error prediction map $E_{c}$, and hidden features $H_{c}$.
Then, the refinement network $G_{\text{refine}}$ employs $H_{c}$, $I$, and $B$
to refine $\alpha_{c}$ and $F^{R}_{c}$ only in regions where the predicted
error $E_{c}$ is large, and produces alpha $\alpha$ and foreground residual
$F^{R}$ at the original resolution. Our model is fully-convolutional and is
trained to work on arbitrary sizes and aspect ratios.

###  4.1 Base Network

The base network is a fully-convolutional encoder-decoder network inspired by
the DeepLabV3 [4] and DeepLabV3+ [5] architectures, which achieved state-of-
the-art performance on semantic segmentation tasks in 2017 and 2018. Our base
network consists of three modules: Backbone, ASPP, and Decoder.

We adopt ResNet-50 [12] for our encoder backbone, which can be replaced by
ResNet-101 and MobileNetV2 [27] to trade-off between speed and quality. We
adopt the ASPP (Atrous Spatial Pyramid Pooling) module after the backbone
following the DeepLabV3 approach. The ASPP module consists of multiple dilated
convolution filters with different dilation rates of 3,6 and 9. Our decoder
network applies bilinear upsampling at each step, concatenated with the skip
connection from the backbone, and followed by a 3$\times$3 convolution, Batch
Normalization [14], and ReLU activation [22] (except the last layer). The
decoder network outputs coarse-grained alpha matte $\alpha_{c}$, foreground
residual $F^{R}_{c}$, error prediction map $E_{c}$ and a 32-channel hidden
features $H_{c}$. The hidden features $H_{c}$ contain global contexts that
will be useful for the refinement network.

###  4.2 Refinement Network

The goal of the refinement network is to reduce redundant computation and
recover high-resolution matting details. While the base network operates on
the whole image, the refinement network operates only on patches selected
based on the error prediction map $E_{c}$. We perform a two-stage refinement,
first at $\frac{1}{2}$ of the original resolution and then at the full
resolution. During inference, we refine $k$ patches, with $k$ either set in
advance or set based on a threshold that trades off between quality and
computation time.

Given the coarse error prediction map $E_{c}$ at $\frac{1}{c}$ of the original
resolution, we first resample it to $\frac{1}{4}$ of the original resolution
$E_{4}$, s.t. each pixel on the map corresponds to a 4$\times$4 patch on the
original resolution. We select the top $k$ pixels with the highest predicted
error from $E_{4}$ to denote the $k$ 4$\times$4 patch locations that will be
refined by our refinement module. The total number of refined pixels at the
original resolution is $16k$.

We perform a two-stage refinement process. First, we bilinearly resample the
coarse outputs, i.e., alpha matte $\alpha_{c}$, foreground residual
$F^{R}_{c}$ and hidden features $H_{c}$, as well as the input image $I$ and
background $B$ to $\frac{1}{2}$ of the original resolution and concatenate
them as features. Then we crop out 8$\times$8 patches around the error
locations selected from $E_{4}$, and pass each through two layers of
3$\times$3 convolution with valid padding, Batch Normalization, and ReLU,
which reduce the patch dimension to 4$\times$4\. These intermediate features
are then upsampled to $8\times 8$ again and concatenated with the 8$\times$8
patches extracted from the original-resolution input $I$ and background $B$ at
the corresponding location. We then apply an additional two layers of
3$\times$3 convolution with valid padding, Batch Normalization and ReLU
(except the last layer) to obtain 4$\times$4 alpha matte and foreground
residuals results. Finally, we upsample the coarse alpha matte $\alpha_{c}$
and foreground residual $F^{R}_{c}$ to the original resolution and swap in the
respective 4$\times$4 patches that have been refined to obtain the final alpha
matte $\alpha$ and foreground residual $F^{R}$. The entire architecture is
illustrated in Figure 2. See appendix for the details of implementation.

###  4.3 Training

All matting datasets provide an alpha matte and a foreground layer, which we
compose onto multiple high-resolution backgrounds. We employ multiple data
augmentation techniques to avoid overfitting and help the model generalize to
challenging real-world situations. We apply affine transformation, horizontal
flipping, brightness, hue, and saturation adjustment, blurring, sharpening,
and random noise as data augmentation to both the foreground and background
layer independently. We also slightly translate the background to simulate
misalignment and create artificial shadows to simulate how the presence of a
subject can cast shadows in real-life environments (see appendix for more
details). We randomly crop the images in every minibatch so that the height
and width are each uniformly distributed between 1024 and 2048 to support
inference at any resolution and aspect ratio.

To learn $\alpha$ w.r.t. ground-truth $\alpha^{*}$, we use an L1 loss over the
whole alpha matte and its (Sobel) gradient:

$\mathcal{L}_{\alpha}=||\alpha-\alpha^{*}||_{1}+||\nabla{\alpha}-\nabla{\alpha^{*}}||_{1}.\vspace{-0.4em}$
(1)

We obtain the foreground layer from predicted foreground residual $F^{R}$,
using $F=\max(\min(F^{R}+I,1),0)$. We compute L1 loss only on the pixels where
$\alpha^{*}>0$:

$\mathcal{L}_{F}=||(\alpha^{*}>0)*(F-F^{*}))||_{1}.\vspace{-0.4em}$ (2)

where that $(\alpha^{*}>0)$ is a Boolean expression.

For refinement region selection, we define the ground truth error map as
$E^{*}=|\alpha-\alpha^{*}|$. We then compute mean squared error between the
predicted error map and the ground truth error map as the loss:

$\mathcal{L}_{E}=||E-E^{*}||_{2}.\vspace{-0.4em}$ (3)

This loss encourages the predicted error map to have larger values where the
difference between the predicted alpha and the ground-truth alpha is large.
The ground-truth error map changes over iterations during training as the
predicted alpha improves. Over time, the error map converges and predicts high
error in complex regions, e.g. hair, that would lead to poor composites if
simply upsampled.

The base network
$(\alpha_{c},F^{R}_{c},E_{c},H_{c})=G_{\text{base}}(I_{c},B_{c})$ operates at
$\frac{1}{c}$ of the original image resolution, and is trained with the
following loss function:

$\mathcal{L}_{\text{base}}=\mathcal{L}_{\alpha_{c}}+\mathcal{L}_{F_{c}}+\mathcal{L}_{E_{c}}.\vspace{-0.4em}$
(4)

The refinement network
$(\alpha,F^{R})=G_{\text{refine}}(\alpha_{c},F^{R}_{c},E_{c},H_{c},I,B)$ is
trained using:

$\mathcal{L}_{\text{refine}}=\mathcal{L}_{\alpha}+\mathcal{L}_{F}.\vspace{-0.4em}$
(5)

We initialize our model’s backbone and ASPP module with DeepLabV3 weights pre-
trained for semantic segmentation on ImageNet and Pascal VOC datasets. We
first train the base network till convergence and then add the refinement
module and train it jointly. We use Adam optimizer and $c=4$, $k=5,000$ during
all the training. For training only the base network, we use batch size 8 and
learning rate [1e-4, 5e-4, 5e-4] for backbone, ASPP, and decoder. When
training jointly, we use batch size 4 and learning rate [5e-5, 5e-5, 1e-4,
3e-4] for backbone, ASPP, decoder, and refinement module respectively.

We train our model on multiple datasets in the following order. First, we
train only the base network $G_{\text{base}}$ and then the entire model
$G_{\text{base}}$ and $G_{\text{refine}}$ jointly on VideoMatte240K, which
makes the model robust to a variety of subjects and poses. Next, we train our
model jointly on PhotoMatte13K to improve the high-resolution details.
Finally, we train our model jointly on Distinctions-646. The dataset has only
362 unique training samples, but it is of the highest quality and contains
human-annotated foregrounds that are very helpful for improving the foreground
quality produced by our model. We omit training on the AIM dataset as a
possible 4th stage and only use it for testing because it causes a degradation
in quality as shown in our ablation study in Section 6.

##  5 Experimental Evaluation

| Alpha | FG
---|---|---
Dataset | Method | SAD | MSE | Grad | Conn | MSE
AIM | DIM${}^{\text{\textdagger}}$ | 37.94 | 80.67 | 32935 | 37861 | -
FBA${}^{\text{\textdagger}}$ | 9.68 | 6.38 | 4265 | 7521 | 1.94
BGM | 16.07 | 21.00 | 15371 | 14123 | 47.98
BGMa | 19.28 | 29.31 | 19877 | 18083 | 42.84
Ours | 12.86 | 12.01 | 8426 | 11116 | 5.31
Distinctions | DIM${}^{\text{\textdagger}}$ | 43.70 | 86.22 | 49739 | 43914 | -
FBA${}^{\text{\textdagger}}$ | 11.03 | 8.32 | 6894 | 9892 | 12.51
BGM | 19.21 | 25.89 | 30443 | 18191 | 36.13
BGMa | 16.02 | 20.18 | 24845 | 14900 | 43.00
Ours | 9.19 | 7.08 | 6345 | 7216 | 6.10
PhotoMatte85 | DIM${}^{\text{\textdagger}}$ | 32.26 | 45.40 | 44658 | 30876 | -
FBA${}^{\text{\textdagger}}$ | 7.37 | 4.79 | 7323 | 5206 | 7.03
BGM | 17.32 | 21.21 | 27454 | 15397 | 14.25
BGMa | 14.45 | 19.24 | 23314 | 13091 | 16.80
Ours | 8.65 | 9.57 | 8736 | 6637 | 13.82
Table 1: Quantitative evaluation on different datasets.
${}^{\text{\textdagger}}$ indicates methods that require a manual trimap.

|  Input |  Ours |  BGM |  BGMa |  Trimap |  FBA |  Trimap |  FBA${}_{\text{auto}}$
---|---|---|---|---|---|---|---|---
|  | Background-based methods | Manual trimap | Segmentation-morph trimap

Figure 4: Qualitative comparison on real images. We produce superior results
at high-resolution with minimal user input. While FBA is competitive, it fails
in a fully automatic application where the segmentation-based trimap is
faulty.

We compare our approach to two trimap-based methods, Deep Image Matting (DIM)
[33] and FBA Matting (FBA) [8], and one background-based method, Background
Matting (BGM) [28]. The input resolution to DIM was fixed at 320$\times$320 by
the implementation, while we set the FBA input resolution to approximately HD
due to memory limits. We additionally train the BGM model on our datasets and
denote it as BGMa (BGM adapted).

Our evaluation uses $c=4$, $k=20,000$ for photos, $c=4$, $k=5,000$ for HD
videos, and $c=8$, $k=20,000$ for 4K videos, where $c$ is the downsampling
factor for the base network and $k$ is the number of patches that get refined.

###  5.1 Evaluation on composition datasets

We construct test benchmarks by separately compositing test samples from AIM,
Distinctions, and PhotoMatte85 datasets onto 5 background images per sample.
We apply minor background misalignment, color adjustment, and noise to
simulate flawed background capture. We generate trimaps from ground-truth
alpha using thresholding and morphological operations. We evaluate matte
outputs using metrics from [26]: MSE (mean squared error) for alpha and
foreground, SAD (sum of absolute difference), Grad (spatial-gradient metric),
and Conn (connectivity) for alpha only. All MSE values are scaled by 103 and
all metrics are only computed over the unknown region of trimaps generated as
described above. Foreground MSE is additionally only measured where the
grouth-truth alpha $\alpha^{*}>0$.

Table 1 shows that our approach outperforms the existing background-based BGM
method across all datasets. Our approach is slightly worse than the state-of-
the-art trimap-based FBA method, which requires carefully annotated manual
trimaps and is much slower than our approach, as shown later in the
performance comparison.

###  5.2 Evaluation on captured data

Although quantitative evaluation on the above-mentioned datasets serves the
purpose of quantifying the performance of different algorithms, it is
important to evaluate how these methods perform on unconstrained real data. To
evaluate on real data, we capture a number of photos and videos containing
subjects in varying poses and surroundings. The videos are captured on a
tripod with consumer smartphones (Samsung S10+ and iPhone X) and a
professional camera (Sony $\alpha$7s II), in both HD and 4K resolution. The
photos are captured in 4000$\times$6000 resolution. We also use some HD videos
presented in the BGM paper that are made public to compare with our method.

For fair comparison in the real-time scenario, where manual trimaps cannot be
crafted, we construct trimaps by morphing segmentation result from DeepLabV3+,
as suggested in [28]. We show results on both trimaps, denoting FBA using this
fully automatic trimap as FBA${}_{\text{auto}}$.

Figure 4 shows our method produces much sharper and more detailed results
around hair and edges compared to other methods. Since our refinement operates
at the native resolution, the quality is far superior relative to BGM, which
resizes the images and only processes them at 512$\times$512 resolution. FBA,
with manual trimap, produces excellent results around hair details, however
cannot be evaluated at resolutions above around HD on standard GPUs. When FBA
is applied on automatic trimaps generated with segmentation, it often shows
large artifacts, mainly due to faulty segmentation.

We extract 34 frames from both the test videos shared by the BGM paper and our
captured videos and photos to create a user study. 40 participants were
presented with an interactive interface showing each input image as well as
the mattes produced by BGM and our approach, in random order. They were
encouraged to zoom in on details and asked to rate one of the mattes as ”much
better”, ”slightly better”, or ”similar”. The results, shown in Table 2,
demonstrate significant qualitative improvement over BGM. 59% of the time
participants perceive our algorithm to be better, compared to 23% for BGM. For
sharp samples in 4K and larger, our method is preferred 75% of the time to
BGM’s 15%.

| Much worse | Worse | Similar | Better | Much better
---|---|---|---|---|---
All | 6% | 17% | 18% | 32% | 27%
4K+ | 5% | 10% | 10% | 34% | 41%
Table 2: User study results: Ours vs BGM

###  5.3 Performance comparison

Table 3 and 4 show that our method is smaller and much faster than BGM. Our
method contains only 55.7% of the parameters compared to BGM. Our method can
achieve HD 60fps and 4K 30fps at batch size 1 on an Nvidia RTX 2080 TI GPU,
considered to be real-time for many applications. It is a significant speed-up
compared to BGM which can only handle 512$\times$512 resolution at 7.8fps. The
performance can be further improved by switching to MobileNetV2 backbone,
which achieves 4K 45fps and HD 100fps. More performance results, such as
adjusting the refinement selection parameter $k$ and using a larger batch
size, are included in the ablation studies and in the appendix.

Method | Backbone | Resolution | FPS | GMac
---|---|---|---|---
FBA |  | HD | 3.3 | 54.3
FBA${}_{\text{auto}}$ |  | HD | 2.9 | 137.6
BGM |  | 5122 | 7.8 | 473.8
Ours | ResNet-50* | HD | 60.0 | 34.3
ResNet-101 | HD | 42.5 | 44.0
MobileNetV2 | HD | 100.6 | 9.9
Ours | ResNet-50* | 4K | 33.2 | 41.5
ResNet-101 | 4K | 29.8 | 51.2
MobileNetV2 | 4K | 45.4 | 17.0
Table 3: Speed measured on Nvidia RTX 2080 TI as PyTorch model pass-through without data transferring at FP32 precision and with batch size 1. GMac does not account for interpolation and cropping operations. For the ease of measurement, BGM and FBA${}_{\text{auto}}$ use adapted PyTorch DeepLabV3+ implementation with ResNet101 backbone as segmentation. Method | Backbone | Parameters | Size
---|---|---|---
FBA |  | 34,693,031 | 138.80 MB
FBA${}_{\text{auto}}$ |  | 89,398,348 | 347.48 MB
BGM |  | 72,231,209 | 275.53 MB
Ours | ResNet-50* | 40,247,703 | 153.53 MB
ResNet-101 | 59,239,831 | 225.98 MB
MobileNetV2 | 5,006,839 | 19.10 MB
Table 4: Model size comparison. BGM and FBA${}_{\text{auto}}$ use DeepLabV3+
with Xception backbone for segmentation.

Natural capture |  BGM |  Ours
---|---|---

Green screen capture |  DaVinci |  Ours
---|---|---

Figure 5: We produce better results than a chroma-keying software, when an
amateur green-screen setup is used.

###  5.4 Practical use

Zoom implementation We build a Zoom plugin that intercepts the webcam input,
collects one no-person (background) shot, then performs real-time video
matting and compositing, streaming the result back into the Zoom call. We test
with a 720p webcam in Linux. The upgrade elicits praise in real meetings,
demonstrating its practicality in a real-world setting.

Comparison to green-screen Chroma keying with a green screen is the most
popular method for creating high-quality mattes. However, it requires even
lighting across the screen and background-subject separation to avoid cast
shadows. In Figure 5, we compare our method against chroma-keying under the
same lighting with an amateur green-screen setup. We find that in the unevenly
lit setting, our method outperforms approaches designed for the green screen.

##  6 Ablation Studies

Role of our datasets We train on multiple datasets, each of which brings
unique characteristics that help our network produce high-quality results at
high-resolution. Table 5 shows the metrics of our method by adding or removing
a dataset from our training pipeline. We find adding the AIM dataset as a
possible 4th stage worsens the metrics even on the AIM test set itself. We
believe it is because samples in the AIM dataset are lower in resolution and
quality compared to Distinctions and the small number of samples may have
caused overfitting. Removal of VideoMatte240K, PhotoMatte13K, and Distinctions
datasets from the training pipeline all result in worse metrics, proving that
those datasets are helpful in improving the model’s quality.

| Alpha | FG
---|---|---
Method | SAD | MSE | Grad | Conn | MSE
Ours* | 12.86 | 12.01 | 8426 | 11116 | 5.31
\+ AIM | 14.19 | 14.70 | 9629 | 12648 | 6.34
\- PhotoMatte13K | 14.05 | 14.10 | 10102 | 12749 | 6.53
\- VideoMatte240K | 15.17 | 17.31 | 11907 | 13827 | 7.04
\- Distinctions | 15.95 | 19.51 | 11911 | 14909 | 14.36
BGM | 16.07 | 21.00 | 15371 | 14123 | 42.84
Table 5: Effect of removing or appending a dataset in the training pipeline,
evaluated on the AIM test set.

Role of the base network We experiment with replacing ResNet-50 with
ResNet-101 and MobileNetV2 as our encoder backbone in the base network. The
metrics in Table 6 show that ResNet-101 has slight improvements over ResNet-50
on some metrics while doing worse on others. This indicates that ResNet-50 is
often sufficient for obtaining the best quality. MobileNetV2 on the other hand
is worse than ResNet-50 on all metrics, but it is significantly faster and
smaller than ResNet-50 as shown in Tables 3 and 4, and still obtains better
metrics than BGM.

Base | Refine | Alpha | FG
---|---|---|---
Backbone | Kernel | SAD | MSE | Grad | Conn | MSE
BGMa |  | 16.02 | 20.18 | 24845 | 14900 | 43.00
MobileNetV2 | 3$\times$3 | 10.53 | 9.62 | 7904 | 8808 | 8.19
ResNet-50* | 3$\times$3 | 9.19 | 7.08 | 6345 | 7216 | 6.10
ResNet-101 | 3$\times$3 | 9.30 | 6.82 | 6191 | 7128 | 7.68
ResNet-50 | 1$\times$1 | 9.36 | 8.06 | 7319 | 7395 | 6.92
Table 6: Comparison of backbones and refinement kernels on the Distinctions
test set

Role of the refinement network Our refinement network improves detail
sharpness over the coarse results in Figure 6, and is effective even in 4K
resolution. Figure 7 shows the effects of increasing and decreasing the
refinement area. Most improvement can be achieved by refining over only 5% to
10% of the image resolution. Table 7 shows that refining only the selected
patches provides significant speedup compared to refining the full image.

|  Input |  480$\times$270 |  HD |  4K
---|---|---|---|---

Figure 6: Effect of refinement, from coarse to HD and 4K. Figure 7: Metrics on the Distinctions test set over percentage of image area refined. Grad and Conn are scaled by 10-3. $k$ | 2,500 | 5,000* | 7,500 | Full
---|---|---|---|---
FPS | 62.9 | 60.0 | 55.7 | 42.8
Table 7: Performance with different $k$ values. Measured on our method with
ResNet-50 backbone at HD.

Patch-based refinement vs. Point-based refinement Our refinement module uses a
stack of 3$\times$3 convolution kernels, creating a 13$\times$13 receptive
field for every output pixel. An alternative is to refine only on points using
1$\times$1 convolution kernels, which would result in a 2$\times$2 receptive
field with our method. Table 6 shows that the $3\times 3$ kernel can achieve
better metrics than point-based kernels, due to a larger receptive field.

Figure 8: Failure cases. Our method fails when the subject casts a substantial
shadow on, or matches color with, the background (top) and when the background
is highly textured (bottom).

Limitations Our method can be used on handheld input by applying homography
alignment to the background on every frame, but it is limited to small motion.
Other common limitations are indicated in Figure 8. We recommend using our
method with a simple-textured background, fixed exposure/focus/WB setting, and
a tripod for the best result.

##  7 Conclusion

We have proposed a real-time, high-resolution background replacement technique
that operates at 4K 30fps and HD 60fps. Our method only requires an input
image and an pre-captured background image, which is easy to obtain in many
applications. Our proposed architecture efficiently refines only the error-
prone regions at high-resolution, which reduces redundant computation and
makes real-time high-resolution matting possible. We introduce two new large-
scale matting datasets that help our method generalize to real-life scenarios.
Our experiment shows our method sets new state-of-the-art performance on
background matting. We demonstrate the practicality of our method by streaming
our results to Zoom and achieve a much more realistic virtual conference call.

Ethics Our primary goal is to enable creative applications and give users more
privacy options through background replacement in video calls. However, we
recognize that image editing can also be used for negative purposes, which can
be mitigated through watermarking and other security techniques in commercial
applications of this work.

## References

  * [1] Yağiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik.  Semantic soft segmentation.  ACM Transactions on Graphics (TOG), 37(4):72, 2018.
  * [2] Yagiz Aksoy, Tunc Ozan Aydin, and Marc Pollefeys.  Designing effective inter-pixel information flow for natural image matting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 29–37, 2017.
  * [3] Shaofan Cai, Xiaoshuai Zhang, Haoqiang Fan, Haibin Huang, Jiangyu Liu, Jiaming Liu, Jiaying Liu, Jue Wang, and Jian Sun.  Disentangled image matting.  International Conference on Computer Vision (ICCV), 2019.
  * [4] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [6] Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang.  Knn matting.  IEEE transactions on pattern analysis and machine intelligence, 35(9):2175–2188, 2013.
  * [7] Yung-Yu Chuang, Brian Curless, David H Salesin, and Richard Szeliski.  A bayesian approach to digital matting.  In CVPR (2), pages 264–271, 2001.
  * [8] Marco Forte and François Pitié.  F,b, alpha matting.  arXiv preprint arXiv:2003.07711, 2020.
  * [9] Eduardo SL Gastal and Manuel M Oliveira.  Shared sampling for real-time alpha matting.  In Computer Graphics Forum, volume 29, pages 575–584. Wiley Online Library, 2010.
  * [10] Minglun Gong and Yee-Hong Yang.  Near-real-time image matting with known background.  In 2009 Canadian Conference on Computer and Robot Vision, pages 81–87. IEEE, 2009.
  * [11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [12] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [13] Qiqi Hou and Feng Liu.  Context-aware image matting for simultaneous foreground and alpha estimation.  International Conference on Computer Vision (ICCV), 2019.
  * [14] S. Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  ArXiv, abs/1502.03167, 2015.
  * [15] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9799–9808, 2020.
  * [16] Anat Levin, Dani Lischinski, and Yair Weiss.  A closed-form solution to natural image matting.  IEEE transactions on pattern analysis and machine intelligence, 30(2):228–242, 2007.
  * [17] Anat Levin, Alex Rav-Acha, and Dani Lischinski.  Spectral matting.  IEEE transactions on pattern analysis and machine intelligence, 30(10):1699–1712, 2008.
  * [18] Yaoyi Li, Qingyao Xu, and Hongtao Lu.  Hierarchical opacity propagation for image matting.  arXiv preprint arXiv:2004.03249, 2020.
  * [19] Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin.  Look into person: Joint body parsing & pose estimation network and a new benchmark.  IEEE transactions on pattern analysis and machine intelligence, 41(4):871–885, 2018.
  * [20] Jinlin Liu, Yuan Yao, Wendi Hou, Miaomiao Cui, Xuansong Xie, Changshui Zhang, and Xian-sheng Hua.  Boosting semantic human matting with coarse annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8563–8572, 2020.
  * [21] Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu.  Indices matter: Learning to index for deep image matting.  International Conference on Computer Vision (ICCV), 2019.
  * [22] V. Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In ICML, 2010.
  * [23] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, E. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, B. Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  ArXiv, abs/1912.01703, 2019.
  * [24] Richard J Qian and M Ibrahim Sezan.  Video background replacement without a blue screen.  In Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), volume 4, pages 143–146. IEEE, 1999.
  * [25] Yu Qiao, Yuhao Liu, Xin Yang, Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei.  Attention-guided hierarchical structure aggregation for image matting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13676–13685, 2020.
  * [26] Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott.  A perceptually motivated online benchmark for image matting.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1826–1833. IEEE, 2009.
  * [27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [28] Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman.  Background matting: The world is your green screen.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2291–2300, 2020.
  * [29] Xiaoyong Shen, Xin Tao, Hongyun Gao, Chao Zhou, and Jiaya Jia.  Deep automatic portrait matting.  In European Conference on Computer Vision, pages 92–107. Springer, 2016.
  * [30] Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum.  Poisson matting.  In ACM Transactions on Graphics (ToG), volume 23, pages 315–321. ACM, 2004.
  * [31] Jingwei Tang, Yagiz Aksoy, Cengiz Oztireli, Markus Gross, and Tunc Ozan Aydin.  Learning-based sampling for natural image matting.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [32] Jue Wang, Michael F Cohen, et al.  Image and video matting: a survey.  Foundations and Trends® in Computer Graphics and Vision, 3(2):97–175, 2008.
  * [33] Ning Xu, Brian Price, Scott Cohen, and Thomas Huang.  Deep image matting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2970–2979, 2017.
  * [34] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 889–898, 2019.
  * [35] Yunke Zhang, Lixue Gong, Lubin Fan, Peiran Ren, Qixing Huang, Hujun Bao, and Weiwei Xu.  A late fusion cnn for digital matting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7469–7478, 2019.
  * [36] Bingke Zhu, Yingying Chen, Jinqiao Wang, Si Liu, Bo Zhang, and Ming Tang.  Fast deep matting for portrait animation on mobile phone.  In Proceedings of the 25th ACM international conference on Multimedia, pages 297–305. ACM, 2017.

##  A Overview

We provide additional details in this appendix. In Sec. B, we describe the
details of our network architecture and implementation. In Sec. C, we clarify
our use of keywords for crawling background images. In Sec. D, we explain how
we train our model and show details of our data augmentations. In Sec. E, we
show additional metrics about our method’s performance. In Sec. F, we show all
the qualitative results used in our user study along with the average score
per sample.

##  B Network

###  B.1 Architecture

Backbone Both ResNet and MobileNetV2 are adopted from the original
implementation with minor modifications. We change the first convolution layer
to accept 6 channels for both the input and the background images. We follow
DeepLabV3’s approach and change the last downsampling block with dilated
convolutions to maintain an output stride of 16. We do not use the multi-grid
dilation technique proposed in DeepLabV3 for simplicity.

ASPP We follow the original implementation of ASPP module proposed in
DeepLabV3. Our experiment suggests that setting dilation rates to (3, 6, 9)
produces the better results.

Decoder

CBR128 - CBR64 - CBR48 - C37

”CBR$k$” denotes $k$ 3$\times$3 convolution filters with same padding without
bias followed by Batch Normalization and ReLU. ”C$k$” denotes $k$ 3$\times$3
convolution filters with same padding and bias. Before every convolution,
decoder uses bilinear upsampling with a scale factor of 2 and concatenates
with the corresponding skip connection from the backbone. The 37-channel
output consists of 1 channel of alpha $\alpha_{c}$, 3 channels of foreground
residual $F^{R}_{c}$, 1 channel of error map $E_{c}$, and 32 channels of
hidden features $H_{c}$. We clamp $\alpha_{c}$ and $E_{c}$ to 0 and 1. We
apply ReLU on $H_{c}$.

Refiner

First stage: | C*BR24 - C*BR16
---|---
Second stage: | C*BR12 - C*4

”C*BR$k$” and ”C*$k$” follow the same definition above except that the
convolution does not use padding.

Refiner first resamples coarse outputs $\alpha_{c}$, $F^{R}_{c}$, $H_{c}$, and
input images $I$, $B$ to $\frac{1}{2}$ resolution and concatenates them as
$[n\times 42\times\frac{h}{2}\times\frac{w}{2}]$ features. Based on the error
prediction $E_{c}$, we crop out top $k$ most error-prone patches $[nk\times
42\times 8\times 8]$. After applying the first stage, the patch dimension
becomes $[nk\times 16\times 4\times 4]$. We upsample the patches with nearest
upsampling and concatenate them with patches at the corresponding location
from $I$ and $B$ to form $[nk\times 22\times 8\times 8]$ features. After the
second stage, the patch dimension becomes $[nk\times 4\times 4\times 4]$. The
4 channels are alpha and foreground residual. Finally, we bilinearly upsample
the coarse $\alpha_{c}$ and $F^{R}_{c}$ to full resolution and replace the
refined patches to their corresponding location to form the final output
$\alpha$ and $F^{R}$.

###  B.2 Implementation

We implement our network in PyTorch [23]. The patch extraction and replacement
can be achieved via the native vectorized operations for maximum performance.
We find that PyTorch’s nearest upsampling operation is much faster on small-
resolution patches than bilinear upsampling, so we use it when upsampling the
patches.

##  C Dataset

VideoMatte240K The dataset contains 484 video clips, which consists a total of
240,709 frames. The average frames per clip is 497.3 and the median is 458.5.
The longest clip has 1500 frames while the shortest clip has 124 frames.
Figure 9 shows more examples from VideoMatte240K dataset.

Figure 9: More examples from VideoMatte240K dataset.

Background The keywords we use for crawling background images are:

airport interior | attic | bar interior
---|---|---
bathroom | beach | city
church interior | classroom interior | empty city
forest | garage interior | gym interior
house outdoor | interior | kitchen
lab interior | landscape | lecture hall
mall interior | night club interior | office
rainy woods | rooftop | stadium interior
theater interior | train station | warehouse interior
| workplace interior |

##  D Training

Table 8 records the training order, epochs, and hours of our final model on
different datasets. We use 1$\times$RTX 2080 TI when training only the base
network and 2$\times$RTX 2080 TI when training the network jointly.

Dataset | Network | Epochs | Hours
---|---|---|---
VideoMatte240K | $G_{\text{base}}$ | 8 | 24
VideoMatte240K | $G_{\text{base}}+G_{\text{refine}}$ | 1 | 12
PhotoMatte13K | $G_{\text{base}}+G_{\text{refine}}$ | 25 | 8
Distinctions | $G_{\text{base}}+G_{\text{refine}}$ | 30 | 8
Table 8: Training epochs and hours on different datasets. Time measured on
model with ResNet-50 backbone.

Additionally, we use mixed precision training for faster computation and less
memory consumption. When using multiple GPUs, we apply data parallelism to
split the minibatch across multiple GPUs and switch to use PyTorch’s
Synchronized Batch Normalization to track batch statistics across GPUs.

###  D.1 Training augmentation

For every alpha and foreground training sample, we rotate to composite with
backgrounds in a ”zip” fashion to form a single epoch. For example, if there
are 60 training samples and 100 background images, a single epoch is 100
images, where the 60 samples first pair with the first 60 background images,
then the first 40 samples pair with the rest of the 40 background images
again. The rotation stops when one set of images runs out. Because the
datasets we use are very different in sizes, this strategy is used to
generalize the concept of an epoch.

We apply random rotation ($\pm$5deg), scale (0.3$\sim$1), translation
($\pm$10%), shearing ($\pm$5deg), brightness (0.85$\sim$1.15), contrast
(0.85$\sim$1.15), saturation (0.85$\sim$1.15), hue ($\pm$0.05), gaussian noise
($\sigma^{2}\leq$0.03), box blurring, and sharpening independently to
foreground and background on every sample. We then composite the input image
using $I=\alpha F+(1-\alpha)B$.

We additionally apply random rotation ($\pm$1deg), translation ($\pm$1%),
brightness (0.82$\sim$1.18), contrast (0.82$\sim$1.18), saturation
(0.82$\sim$1.18), and hue ($\pm$0.1) only on the background 30% of the time.
This small misalignment between input $I$ and background $B$ increases model’s
robustness on real-life captures.

We also find creating artificial shadows increases model’s robustness because
subjects in real-life often cast shadows on the environment. Shadows are
created on $I$ by darkening some areas of the image behind the subject
following the subject’s contour 30% of the time. Examples of composited images
are shown in Figure 10. The bottom row shows examples of shadow augmentation.

Figure 10: Training samples with augmentations. Bottom row are samples with
shadow augmentation. Actual samples have different resolutions and aspect
ratios.

###  D.2 Testing augmentation

For AIM and Distinctions, which have 11 human test samples each, we pair every
sample with 5 random backgrounds from the background test set. For
PhotoMatte85, which has 85 test samples, we pair every sample with only 1
background. We use the method and metrics described in [26] to evaluate the
resulting sets of 55, 55, and 85 images.

We apply a random subpixel translation ($\pm$0.3 pixels), random gamma
(0.85$\sim$1.15), and gaussian noise ($\mu=\pm 0.02$, $0.08\leq\sigma^{2}\leq
0.15$) to background $B$ only, to simulate misalignment.

The trimaps used as input for trimap-based methods and for defining the error
metric regions are obtained by thresholding the grouth-truth alpha between
0.06 and 0.96, then applying 10 iterations of dilation followed by 10
iterations of erosion using a 3$\times$3 circular kernel.

##  E Performance

Table 9 shows the performance of our method on two Nvidia RTX 2000 series
GPUs: the flagship RTX 2080 TI and the entry-level RTX 2060 Super. The entry-
level GPU yields lower FPS but is still within an acceptable range for many
real-time applications. Additionally, Table 10 shows that switching to a
larger batch size and a lower precision can increase the FPS significantly.

Figure 11: The web UI for our user study. Users are shown the original image and two result images from Ours and BGM methods. Users are given the instruction to rate whether one algorithm is ”much better”, ”slightly better”, or both as ”similar”. GPU | Backbone | Reso | FPS
---|---|---|---
RTX 2080 TI | ResNet-50 | HD | 60.0
4K | 33.2
MobileNetV2 | HD | 100.6
4K | 45.4
RTX 2060 Super | ResNet-50 | HD | 42.8
4K | 23.3
MobileNetV2 | HD | 75.6
4K | 31.3
Table 9: Performance on different GPUs. Measured with batch size 1 and FP32 precision. Backbone | Reso | Batch | Precision | FPS
---|---|---|---|---
MobileNetV2 | HD | 1 | FP32 | 100.6
8 | FP32 | 138.4
8 | FP16 | 200.0
4K | 8 | FP16 | 64.2
Table 10: Performance using different batch sizes and precisions. Measured on
RTX 2080 TI.

##  F Additional Results

In Figures 12, 13, 14, we show all 34 examples in the user study, along with
their average rating and results by different methods. Figure 11 shows the web
UI for our user-study.

|  +2.0
---|---
-3.3
+1.8
+0.1
+3.3
+9.0
+7.5
+7.1
-2.6
-1.3
-0.8
2.4

|  Input |  Ours |  BGM |  BGMa |  Trimap |  FBA |  Trimap |  FBA${}_{\text{auto}}$ | Score
---|---|---|---|---|---|---|---|---|---
|  | Background-based methods | Manual trimap | Segmentation-morph trimap |

Figure 12: Additional qualitative comparison (1/3). Average user ratings
between Ours and BGM are included. A score of -10 denotes BGM is ”much
better”, -5 denotes BGM is ”slightly better”, 0 denotes ”similar”, +5 denotes
Ours is ”slightly better”, +10 denotes Ours is ”much better”. Our method
receives an average 3.1 score.

|  -5.5
---|---
+2.9
+2.3
+3.8
+3.6
-3.6
-2.4
+8.8
+6.0
+5.6
+6.3

|  Input |  Ours |  BGM |  BGMa |  Trimap |  FBA |  Trimap |  FBA${}_{\text{auto}}$ | Score
---|---|---|---|---|---|---|---|---|---
|  | Background-based methods | Manual trimap | Segmentation-morph trimap |

Figure 13: Additional qualitative comparisons (2/3)

|  +7.4
---|---
+8.8
+7.4
+7.0
-3.5
+6.8
-3.3
+5.8
+5.1
+0.1
+3.8

|  Input |  Ours |  BGM |  BGMa |  Trimap |  FBA |  Trimap |  FBA${}_{\text{auto}}$ | Score
---|---|---|---|---|---|---|---|---|---
|  | Background-based methods | Manual trimap | Segmentation-morph trimap |

Figure 14: Additional qualitative comparisons (3/3)


aainstitutetext: Department of Mathematics, Statistics and Physics, Punjab
Agricultural University, Ludhiana-141004, Indiabbinstitutetext: Mitchell
Institute for Fundamental Physics and Astronomy, Department of Physics and
Astronomy, Texas A&M University, College Station, TX 77843,
USAccinstitutetext: Department of Physics, University of Seoul, Seoul 02504,
Republic of Korea(*)(*)institutetext: Now at Northwestern University

# Probing bottom-associated production of a TeV scale scalar decaying to a top
quark and dark matter at the LHC

Amandeep Kaur Kalsi b Teruki Kamon c Seulgi Kim c Jason S.H. Lee b Denis
Rathjens c Youn Jung Roh b(*) Adrian Thompson c Ian James Watson

###### Abstract

A minimal non-thermal dark matter model that can explain both the existence of
dark matter and the baryon asymmetry in the universe is studied. It requires
two color-triplet, iso-singlet scalars with $\mathcal{O}$(TeV) masses and a
singlet Majorana fermion with a mass of $\mathcal{O}$(GeV). The fermion
becomes stable and can play the role of the dark matter candidate. We consider
the fermion to interact with a top quark via the exchange of QCD-charged
scalar fields coupled dominantly to third generation fermions. The signature
of a single top quark production associated with a bottom quark and large
missing transverse momentum opens up the possibility to search for this type
of model at the LHC in a way complementary to existing monotop searches.

††preprint:  MI-HET-803

##  1 Introduction

The puzzle of the observed matter-antimatter asymmetry in the universe has
been a primary catalyst in the search for physics beyond the standard model of
particle physics. In particular, models that violate baryon number $B$ are
necessary, since this violation is among the three Sakharov conditions
Sakharov:1991  necessary to accommodate the observed baryon asymmetry, the
other two being $CP$-violation and the thermal inequilibrium. This leaves
$B$-violation a robustly-motivated property of many beyond-the-Standard Model
(BSM) models and searches for those at the Large Hadron Collider (LHC).

Baryon asymmetry and the existence of Dark Matter (DM) may be connected
through the fact that the energy densities of DM and baryonic matter in the
universe differ by only a factor of 5 – a small difference in light of the
separation of energy and mass scales that have already been observed in nature
Arkani-Hamed:2000ifx ; PhysRevLett.82.896 ; DES:2018paw ; Planck:2018vyg .
Using this coincidence of scales as a theoretical guide, concrete models that
support both the observed DM relic densities and baryogenesis have been
vigorously investigated in many contexts.

In particular, BSM models with $\Delta B=2$ operators that can lie in the TeV
range of QCD charged scalars have been of interest for their connection to DM
Andrea:2011ws ; Agram:2013wda ; Alvarez:2013jqa ; Wang:2011uxa ; Kumar:2013jgb
; Blanke:2020bsf , predicting resonantly-produced monotop final states in
reach of LHC experiments. This is due to their simple event topology that
occurs when a single top quark is produced in association with missing
transverse momentum, which could be a signature of the production of DM
particles. It also enables distinguishing left-chiral from right-chiral
couplings Allahverdi:2015mha . This type of search can help to probe the
aforementioned varieties of DM models and potentially shed light on neutron-
anti-neutron ($n$-$\bar{n}$) oscillations, baryogenesis, non-thermal DM
production, and baryon-number-violating sectors in the broader sense (see also
refs. Alonso-Alvarez:2021qfd ; Racker:2014yfa ).

Several searches have already been performed at the Tevatron and the LHC with
center-of-mass energies $\sqrt{s}=1.96$ TeV Aaltonen:2012ek , 8 TeV
Khachatryan:2014uma ; Aad:2014wza , and $13$ TeV Sirunyan:2018gka  to probe
new physics associated with baryon number violation. For example, scalars
($\phi$) produced via $\bar{d}_{i}\bar{d}_{j}\rightarrow\phi\rightarrow t\psi$
are excluded for masses below 3.4 TeV in events with large missing transverse
momentum and a hadronically-decaying top quark in ref. Sirunyan:2018gka ,
assuming the same couplings for $\phi$ production in $d$-$s$, $s$-$b$ and
$b$-$d$ quark fusions. Here $\psi$ is a DM fermion, and $i$ and $j$
($i\not=j$) are the flavor indices of down-type quark. It should be noted that
the limit of 3.4 TeV is not applicable if the coupling being responsible for
the $d$-$s$ fusion is zero.

One particular connection to these models is made through a variety of $\Delta
B=2$ models which can be tested at the LHC. These models may contain novel
scalar sectors including color-charged and electroweak-charged scalars that
couple DM candidates to quarks Fajfer:2020tqf , and the possibility of a non-
trivial flavor structure in these interactions can also be used to probe
$n$-$\bar{n}$ oscillations and explain the neutron decay anomaly
Fridell:2021gag ; Chen:2022gjd ; Fornal:2018eol . Since DM particles with
masses of around 1 GeV can be probed at the LHC in connection with monotop
signals, these search results can be interpreted as sensitivity to mirror
neutrons and mirror sector DM Tan:2023mpj . Because of the far reaching
implications of baryon number violation to both the cosmic history and to the
nature of modern particle physics puzzles, the search for monotop-type signals
at the LHC could lead to new discoveries in both the fields of particle
physics and cosmology Allahverdi:2010im ; Allahverdi:2013mza .

In this paper, we devise a search for new scalars in single top quark events
with the requirement of an associated bottom quark. Thereby, we specifically
probe color-triplet scalars dominantly-coupled to third generation fermions as
a benchmark test for the LHC sensitivity to baryon-number-violating DM
interactions. Summarily, we explore the implications for baryogenesis,
$n$-$\bar{n}$ oscillations, and the DM and baryon coincidence puzzles.
Experimentally, this has the desirable property of being orthogonal in
selection to prior monotop searches Aad:2014wza ; Sirunyan:2018gka  that veto
additional $b$-tagged jets to suppress $t\bar{t}$ background for their monotop
searches.

##  2 The Model and its Collider Signals

The $\Delta B=2$ baryon-number-violating model that we consider features two
charged scalars $X_{\alpha}$, $\alpha=1,2$, which are color-triplets under
$SU(3)_{c}$, singlets under $SU(2)_{L}$, and have hypercharge $+4/3$. This
scalar allows for operators such as $X_{\alpha}\psi u^{c}_{i}$ and
$X_{\alpha}^{*}d_{i}^{c}d_{j}^{c}$, which can support baryogenesis by
interference of the tree-level and loop-level decays of $X\to d_{i}d_{j}$
Allahverdi:2010im ; Allahverdi:2013mza ; Allahverdi:2017edd ; Dutta:2014kia .
Here $i$ and $j$ ($i\not=j$) are the quark flavor indices. (The color indices
are omitted.) In the Weyl representation, our minimal interaction Lagrangian
reads

$\mathcal{L}\supset\lambda_{\alpha i}X_{\alpha}\psi u^{c}_{i}+\lambda_{\alpha
ij}^{\prime}X_{\alpha}^{*}d_{i}^{c}d_{j}^{c}+\frac{m_{\psi}}{2}\bar{\psi}^{c}\psi+\text{h.c.}$
(1)

where “h.c.” refers to the Hermitian conjugate of the preceding terms in the
Lagrangian. We are interested in single top quark and $X_{\alpha}$ production
($X_{\alpha}\psi t^{c}$) from $d$-$b$ and $s$-$b$ fusion as in Figure 1. In
addition, to suppress the loop-level interference terms, we work with the
assumption that $m_{X_{1}}\ll m_{X_{2}}$. After focusing on the couplings to
$t$, $s$, and $b$ quarks, the phenomenological part of the Lagrangian relevant
for our associated jet search becomes simply

$\mathcal{L}_{\textrm{single top}}\supset\lambda_{\psi t}X_{1}\psi
t^{c}+\lambda_{db}^{\prime}X_{1}^{*}d^{c}b^{c}+\lambda_{sb}^{\prime}X_{1}^{*}s^{c}b^{c}+\text{h.c.}$
(2)

To further indicate the production of $X_{1}$ through $\lambda_{db}^{\prime}$
or $\lambda_{sb}^{\prime}$ experimentally, we require an additional initial
state radiation (ISR) $b$ quark from gluon splitting as shown in Figure 1.
Relabeling the coupling matrix indices for this specific production mode:
$\lambda_{db}^{\prime}$ or $\lambda_{sb}^{\prime}$ $\to\lambda_{1}$, and
$\lambda_{\psi t}\to\lambda_{2}$ for the sake of clarity, the cross section
for $X$ production with $\mathcal{O}$(TeV) masses is determined by
$\lambda_{1}$ and $\lambda_{2}$ as follows:

$\mathcal{\sigma}\propto\frac{|\lambda_{1}|^{2}|\lambda_{2}|^{2}}{2|\lambda_{1}|^{2}+|\lambda_{2}|^{2}}$
(3)

While there is a vertex factor from the gluon splitting suppressing this
production mode, the steep falloff of the bottom parton distribution function
(PDF) prefers a gluon parton initial state instead for heavy $X_{\alpha}$,
lending itself to branching fraction of 24.8% for a mass of 1 TeV, 25.2% for
1.5 TeV, and 25.6% for 2 TeV at $(\lambda_{1},\lambda_{2})=(0.1,0.1)$.

$b$$\bar{d}$$\psi$$b$$\ell^{+}$$\nu_{\ell}$$g$$\bar{b}$$X$$t$$W^{+}$ Figure 1:
A representative Feynman diagram for the signal process with an additional ISR
$b$ quark from gluon splitting. Here $\psi$ is a fermionic DM candidate.
Figure 2: Mass distribution at truth-level for $X_{1}$ as a function of
($\lambda_{1},\lambda_{2}$) for $m_{X_{1}}=1$ TeV. The decay width is
proportional to $2|\lambda_{1}|^{2}+|\lambda_{2}|^{2}$. The $x$-axis of each
subplot is in the range of [800, 1200] GeV. All the subplots are normalized to
the total event in each case. cross section $\sigma$ [fb]
---
$\lambda_{1}$ | $\lambda_{2}$
0.1 | 0.2 | 0.5 | 1.0
0.1 | 2.22 | 5.42 | 9.12 | 10.3
0.2 | 2.40 | 8.13 | 25.5 | 36.3
0.5 | 2.47 | 9.62 | 51.1 | 132
1.0 | 2.57 | 10.3 | 61.1 | 210
Table 1: Cross sections for $m_{X_{1}}=1$ TeV. The cross section is inversely
proportional to $2|\lambda_{1}|^{2}+|\lambda_{2}|^{2}$.

##  3 Simulation

MadGraph5_aMC@NLO (v2.6.7) Alwall:2014hca  has been used for the generation of
the single top quark plus bottom quark signal events, as well as all
background samples listed from this point onward, in proton-proton collisions
at $\sqrt{s}=13$ TeV. Parton showering was performed by PYTHIA8 (v8.244)
SJOSTRAND2015159 .

Up to two additional initial state ISR/final state radiation (FSR) jets,
excluding one ISR $b$-tagged jet and one FSR $b$-tagged jet, are included in
the generation. The analysis is restricted to leptonic top-quark decays ($t\to
Wb\to l\nu b$), where $l$ is a muon or an electron. The mass of $X_{1}$ is set
to 1 TeV, 1.5 TeV or 2 TeV, of $X_{2}$ to 10 TeV, and of $\psi$ to 1 GeV.
Samples were generated with an $11\times 9$ grid of
$\\{\lambda_{1},\lambda_{2}\\}$ with $\lambda_{1}$ in range of [0.01, 1.0] and
$\lambda_{2}$ in range of [0.05, 1.0] for $m_{X_{1}}$ = 1 TeV, and an
$11\times 5$ grid of $\\{\lambda_{1},\lambda_{2}\\}$ with $\lambda_{1}$ in
range of [0.01, 1.0] and $\lambda_{2}$ in range of [0.1, 1.0] for 1.5 and 2
TeV. We set the upper limit to 1.0 to constrain the width of $X$ within the
resonant regime. The distributions of $m_{X1}$ as a function of
$\\{\lambda_{1},\lambda_{2}\\}$ are shown in Figure 2.

We have generated background samples for $W(\to l\nu)$+jets (up to additional
two $b$-tagged jets), $Z(\to l^{+}l^{-})$+jets (up to 2 jets, $b/\overline{b}$
instead of j), single top quark, semileptonic $t\bar{t}$, and diboson ($WW$,
$WZ(\to l\nu b\overline{b})$, $ZZ$) events. We generated one million events
for each single top quark, $WW$ and $ZZ$ sample. Other backgrounds were
generated in 8 bins of $H_{\textrm{T}}$ with one million events per bin. All
samples were simulated at leading order (LO).

A CMS-like detector simulation was performed using DELPHES
(v3.4.2)deFavereau:2013fsa  with a modified DELPHES card. In the leptonic top
quark decays, the lepton is difficult to resolve due to the boost of the top
quark from the $X$ decay causing the lepton to be included in the bottom jet
during the jet clustering. Therefore, we have modified the standard CMS
DELPHES card to remove isolation. The anti-$k_{\textrm{T}}$ jet algorithm
Cacciari_2008  is used with jet radius $R$ = 0.4 to identify jets. The
$b$-tagging efficiency has been also updated by emulating the efficiency for
the medium working point of the CSVv2 algorithm CMS-PAS-BTV-15-001 ;
BTV-16-002 .

##  4 Object and event selection

We define in this chapter a set of baseline selections and object definitions
before applying any machine-learning, when the section is explicitly not about
machine learning.

In signal events, we expect to have one $b$-tagged jet originating from gluon
splitting on the one hand, and a top quark decaying to another $b$-tagged jet,
one lepton ($e$ or $\mu$) and a neutrino on the other hand. Additionally, a
large amount of missing transeverse momentum $p_{\textrm{T}}^{\textrm{miss}}$
is expected, due to the DM candidate $\phi$ and the neutrino. Therefore, we
select events with at least two $b$-tagged jets. They are required to have
${|\eta|<}$ 2.4 and $p_{\textrm{T}}>$ 50(30) GeV for the (sub-)leading
$b$-tagged jet. Additional jets are allowed. Furthermore, we select
$p_{\textrm{T}}^{\textrm{miss}}>$ 50 GeV and exactly one light lepton ($e$ or
$\mu$) with $p_{\textrm{T}}>$ 30 GeV and ${|\eta|<}$ 2.5(2.4) for electrons
(muons) without isolation requirements. We expect the top quark from the $X$
decay to be boosted in our model, as $X$ is far heavier than the top quark.
This often causes the leading lepton to be clustered into the leading
$b$-tagged jet. As the proportion of transverse momentum within typical jets
carried by light leptons is small (typically less than $5\%$, see e.g.
CMS:2016lmd ), we don’t expect distinguishing the high $p_{\textrm{T}}$
leptons from most non-prompt leptons to be an issue. To avoid misjudging the
top-quark-originated leading $b$-tagged jet’s direction and overestimating its
momentum, we subtract the highest $p_{\textrm{T}}$ lepton in the event from
its jets, whenever it is a constituent of the jet. We define leading
$b$-tagged jet $b_{1}$ and sub-leading $b$-tagged jet $b_{2}$ after this
subtraction, where their $p_{\textrm{T}}$ values satisfy our baseline
selection of 50 and 30 GeV, respectively.

The $\Delta R(b_{1},l)$ distribution in Figure 3 shows that the leading lepton
points at the rim of the jet when it is clustered unlike high-momentum leptons
from hadronization. They tend to originate in the center of jet, where most of
the energy during hadronization is deposited. Thus, the resulting angular
difference from distinguishing the boosted lepton has good background
separation power. Not only $b_{2}$ and $X$ are back-to-back due to momentum
conservation, but $X$ is essentially at the rest in the laboratory frame, so
its decay products also have maximal separation in $\Delta\phi$. Consequently,
as $p_{\textrm{T}}^{\textrm{miss}}$ is dominated by the $\phi$ and neutrino
contributions, it also exhibits a large angular distance in the transeverse
plane to $b_{2}$, as shown in Figure 3 and serves well for background
separation. These effects are the more pronounced, the heavier $X$ is.

The significance with baseline selections is 0.11 (0.016, 0.0033) for
$m_{X_{1}}$ = 1 (1.5, 2) TeV sample at $(\lambda_{1},\lambda_{2})=(0.1,0.1)$.
In this analysis, significance is defined as ${n_{s}/\sqrt{n_{b}}}$, where
$n_{s}$ and $n_{b}$ are the number of signal and background events,
respectively.

Figure 3: $\Delta R$($b_{1}$, $l$) (left) and
$\Delta\phi(b_{2},p_{\textrm{T}}^{\textrm{miss}})$ (right) distributions after
applying baseline selections. We expect that the leading $b$-tagged jet
($b_{1}$) and the selected lepton are from the top decay and $b_{2}$ is from
gluon splitting. As the top quark is more boosted, $\Delta R$($b_{1}$, $l$)
becomes smaller. Moreover, $b_{2}$ likely appears in back-to-back to
$p_{\textrm{T}}^{\textrm{miss}}$ where it represents a vector sum of momenta
of the neutrino from the top quark decay and the DM particle.

##  5 BDT training

Boosted decision trees (BDTs), provided by the Toolkit for Multivariate Data
Analysis with ROOT Hoecker_2007 , are used to combine the discriminating power
of several kinematic distributions into a single classifier in order to
further improve the separation between signal and backgrounds. The BDT
training is performed for events after our baseline selections in signal
samples with $\lambda_{1}=\\{0.1,0.3,1.0\\}$ and
$\lambda_{2}=\\{0.1,0.3,1.0\\}$ and the background samples mentioned in
Section 3. A separate BDT was trained for each signal mass point with the
combined signal samples.

The following input variables were used in training: $p_{\textrm{T}}$ and
$\eta$ of the $b_{1}$, $b_{2}$, $l$, and $b_{1}+l$, $\Delta R(b_{1},b_{2})$,
$\Delta R(b_{1},l)$, $\Delta\phi(b_{2},p_{\textrm{T}}^{\textrm{miss}})$,
$\Delta\phi(l,p_{\textrm{T}}^{\textrm{miss}})$,
$m_{\textrm{T}}(b_{1},p_{\textrm{T}}^{\textrm{miss}})$, and
$m_{\textrm{T}}(l,p_{\textrm{T}}^{\textrm{miss}})$. To get the best
performance, an optimization of the BDT hyperparameter on NTree and MaxDepth
was conducted. (NTrees, MaxDepth) = (3000, 5) for $m_{X_{1}}$ = 1 TeV and
(2500, 5) for 1.5 TeV and 2 TeV were chosen as the optimized parameters. The
variable with the largest variable importance value was
$\Delta\phi(b_{2},p_{\textrm{T}}^{\textrm{miss}})$, as shown in the right plot
of Figure 3. The model response and receiver operating characteristic (ROC)
curve of the BDT with the optimized hyperparameters are shown in Figure 4,
which shows that tail becomes shorter as $m_{X_{1}}$increases, which means
that signal and background separate more. The area under the curve (AUC) of
the ROC curve for $m_{X_{1}}$ = 1 (1.5, 2) TeV is 0.989 (0.994, 0.995).

Figure 4: The model response of the BDT for $m_{X_{1}}$ of 1 (top left), 1.5
(top right), and 2 TeV (bottom left) case and the ROC curve of the BDT for
$m_{X_{1}}$ of 2 TeV (bottom right) with the optimized BDT hyperparameters.

In addition to the baseline selections as defined in Section 4, we require
that events have BDT $>\alpha$, where $\alpha$ is optimized to produce the
largest signal significance. We chose the value of $\alpha$ as 0.1425 (0.1900,
0.2100) for $m_{X_{1}}$ = 1 (1.5, 2) TeV scenario. Figure 5 shows significance
and the number of events remaining versus $\alpha$ for
$(\lambda_{1},\lambda_{2})=(0.1,0.3)$, $m_{X_{1}}$ = 1 TeV signal samples.

Figure 5: Significance and the number of events remaining as a function of the
BDT discriminator with $m_{X_{1}}$ = 1 TeV, and
$(\lambda_{1},\lambda_{2})=(0.1,0.3)$.

##  6 Results

We evaluate the expected exclusion limits on three benchmark scenarios with
$m_{X_{1}}$ = 1, 1.5, and 2 TeV in our model on
$\\{\lambda_{1},\lambda_{2}\\}$ space for each mass point, assuming integrated
luminosity scenarios of 300 fb-1 and 3000 fb-1 corresponding to the projected
Run 3 and Phase 2 LHC luminosities, respectively. The feasibility is computed
in the $\\{\lambda_{1},\lambda_{2}\\}$ space. RooStats Moneta:2010pm  is used
to perform the statistical analysis. We create a model based on Poisson
statistics

$f(N_{b};N_{s}+N_{b})=\frac{{(N_{s}+N_{b})^{N_{b}}}e^{-(N_{s}+N_{b})}}{N_{b}!}$
(4)

while considering the systematic uncertainty as Gaussian-distributed, taking

$g(N_{b})=e^{-\frac{(N_{b}-N_{b_{0}})^{2}}{2{\sigma}^{2}}}$ (5)

Here, $N_{s}$ is the expected number of signal events, $N_{b}$ is the expected
number of background events which has an uncertainty of $\sigma_{b}$ and
$N_{b_{0}}$ is the estimated number of background events after applying the
baseline selections and cut on BDT output variable. The model probability
density function is defined as:

$f(N_{b};N_{s}+N_{b})\cdot g(N_{b})$ (6)

We used a uniform distribution as a prior probability density function on the
number of signal events.

Limits on $\\{\lambda_{1},\lambda_{2}\\}$ are computed using RooStats
BayesianCalculator. It computes the posterior probability based on the Poisson
likelihood in Equation 4. Figure 6 shows the 95% confidence level exclusion
and the 5$\sigma$ discovery potential of $X_{1}$ with masses of 1, 1.5, and 2
TeV for integrated luminosities of 300 fb-1 expected by the end of Run 3 and
3000 fb-1 expected by the end of High Luminosity LHC era. At 300 fb-1, we
expect this type of analysis could constrain $\lambda_{2}$ which is related to
the top quark coupling to at most 0.1 (0.15, 0.25) for $\lambda_{1}>0.1$ in
the $m_{X_{1}}$ = 1 (1.5, 2) TeV case in 95% CL. Furthermore, $\lambda_{2}$
could be constrained to 0.09 (0.1, 0.18) for $\lambda_{1}>0.1$ at 3000 fb-1.

At the hadronization scale, the non-perturbative effects are expected to be
large, leading to the unreliable calculation from MadGraph5_aMC@NLO. This
lower limit of $\lambda_{i}$ following perturbative QCD theory bounds its
lifetime within the hadronization scale. We marked the region of $m_{X_{1}}$ =
1 TeV which suffers from this issue by a solid area in Figure 6. The limits of
the other two scenarios are not visualized in Figure 6 since it’s not affected
on the limit.

Higher masses of $X_{1}$ are more challenging to exclude, as the cross section
decreases, albeit the branching fraction of additional $b$-quarks from ISR
gluon splitting has a trend in the opposite direction as tree level
contributions of the $b$ PDF diminish faster with increasing energy
requirements than the gluon PDF scaling does, while the additional vertex
factor for the splitting remains constant. The limits presented here will be
improved in combination with an analysis of events with hadronically-decaying
top quarks, probing the interesting region of $\lambda_{i}\approx{\cal
O}(0.1)$ Allahverdi:2017edd .

Figure 6: Limits on $\\{\lambda_{1},\lambda_{2}\\}$ at 95% confidence level
and 5$\sigma$ discovery potential for $m_{X_{1}}$ = 1.0, 1.5 and 2.0 TeV at
300 fb-1 (left) and 3000 fb-1 (right). Regions above the contours are expected
to be excluded.

##  7 Summary

We have investigated the potential of searching for baryon-number-violating DM
candidates in association with a bottom jet, missing transverse momentum, and
a top quark at the LHC. The DM candidate in this model is a singlet Majorana
fermion that interacts with top quarks via the exchange of QCD-charged scalar
$X_{1}$. The applicability of the analysis is limisted to light DM particles
($m_{\phi}<<m_{X_{1}}$). The additional ISR $b$ quark was required to
distinguish production modes of $X_{1}$ via $\lambda_{db}^{\prime}$ or
$\lambda_{sb}^{\prime}$ experimentally from previous search strategies. The
95% upper limits and $5\sigma$ limits on couplings are shown for $m_{X_{1}}=$
1, 1.5, and 2 TeV at an integrated luminosity of 300 fb-1 expected by the end
of Run 3 and 3000 fb-1 by the end of High Luminosity LHC era. This reach is
based on a novel approach requiring an associated $b$-tagged jet and utilizing
machine-learning techniques. Traditional monotop analyses veto additional
$b$-tagged jets to suppress backgrounds. We therefore show that the inclusion
of the second $b$-tagged jet offers a complementary analysis design with
comparable reach to search for DM candidates with baryon-number-violating
interactions. These are of particular importance, given they could explain
both the observed DM abundance as well as the baryon asymmetry in the
universe, simultaneously. Our analysis utilized a leptonic top quark decay
while e.g. the 13 TeV CMS analysis Sirunyan:2018gka  utilizes boosted hadronic
top quark identification, so there is further room given the larger branching
ratio of the hadronic decay for possible improvement with another dedicated
hadronic analysis.

## Acknowledgements

We thank B. Dutta for the motivating discussions and guidance in the model
selection and search strategy. T.K. D.R. and A.T. are supported in part by the
US Department of Energy grant DE-SC0010813. S.K. and J.L. are supported by the
National Research Foundation of Korea (NRF) grants No. 2023R1A2C2002751. Y.R.
and I.W. are supported by NRF grants No.2021R1A2C1093704 and
2018R1A6A1A06024977, respectively.

## References

  * (1) A. D. Sakharov, _Violation of CP in variance, C asymmetry, and baryon asymmetry of the universe_ , _Phys. Usp._ 34 (1991) 392–393.
  * (2) N. Arkani-Hamed, L. J. Hall, C. F. Kolda and H. Murayama, _A New perspective on cosmic coincidence problems_ , _Phys. Rev. Lett._ 85 (2000) 4434–4437, [astro-ph/0005111].
  * (3) I. Zlatev, L. Wang and P. J. Steinhardt, _Quintessence, cosmic coincidence, and the cosmological constant_ , _Phys. Rev. Lett._ 82 (1999) 896–899.
  * (4) DES Collaboration, T. M. C. Abbott et al., _First Cosmology Results using Type Ia Supernovae from the Dark Energy Survey: Constraints on Cosmological Parameters_ , _Astrophys. J. Lett._ 872 (2019) L30, [1811.02374].
  * (5) Planck Collaboration, N. Aghanim et al., _Planck 2018 results. VI. Cosmological parameters_ , _Astron. Astrophys._ 641 (2020) A6, [1807.06209].
  * (6) J. Andrea, B. Fuks and F. Maltoni, _Monotops at the LHC_ , _Phys. Rev. D_ 84 (2011) 074025, [1106.6199].
  * (7) J.-L. Agram, J. Andrea, M. Buttignol, E. Conte and B. Fuks, _Monotop phenomenology at the Large Hadron Collider_ , _Phys. Rev. D_ 89 (2014) 014028, [1311.6478].
  * (8) E. Alvarez, E. Coluccio Leskow, J. Drobnak and J. F. Kamenik, _Leptonic Monotops at LHC_ , _Phys. Rev. D_ 89 (2014) 014016, [1310.7600].
  * (9) J. Wang, C. S. Li, D. Y. Shao and H. Zhang, _Search for the signal of monotop production at the early LHC_ , _Phys. Rev. D_ 86 (2012) 034008, [1109.5963].
  * (10) A. Kumar, J. N. Ng, A. Spray and P. T. Winslow, _Tracking down the top quark forward-backward asymmetry with monotops_ , _Phys. Rev. D_ 88 (2013) 075012, [1308.3712].
  * (11) M. Blanke, P. Pani, G. Polesello and G. Rovelli, _Single-top final states as a probe of top-flavoured dark matter models at the LHC_ , _JHEP_ 01 (2021) 194, [2010.10530].
  * (12) R. Allahverdi, M. Dalchenko, B. Dutta, A. Flórez, Y. Gao, T. Kamon et al., _Distinguishing Standard Model Extensions using Monotop Chirality at the LHC_ , _JHEP_ 12 (2016) 046, [1507.02271].
  * (13) G. Alonso-Álvarez, G. Elor and M. Escudero, _Collider signals of baryogenesis and dark matter from B mesons: A roadmap to discovery_ , _Phys. Rev. D_ 104 (2021) 035028, [2101.02706].
  * (14) J. Racker, _Mini-review on baryogenesis at the TeV scale and possible connections with dark matter_ , _Nucl. Part. Phys. Proc._ 273-275 (2016) 334–339, [1410.5482].
  * (15) CDF Collaboration, T. Aaltonen et al., _Search for a dark matter candidate produced in association with a single top quark in $p\bar{p}$ collisions at $\sqrt{s}=1.96$ TeV_, _Phys. Rev. Lett._ 108 (2012) 201802, [1202.5653].
  * (16) CMS  Collaboration, V. Khachatryan et al., _Search for Monotop Signatures in Proton-Proton Collisions at $\sqrt{s}=$ 8 TeV_, _Phys. Rev. Lett._ 114 (2015) 101801, [1410.1149].
  * (17) ATLAS Collaboration, G. Aad et al., _Search for invisible particles produced in association with single-top-quarks in proton-proton collisions at $\sqrt{s}$ = 8 TeV with the ATLAS detector_, _Eur. Phys. J. C_ 75 (2015) 79, [1410.5404].
  * (18) CMS Collaboration, A. M. Sirunyan et al., _Search for dark matter in events with energetic, hadronically decaying top quarks and missing transverse momentum at $\sqrt{s}=13$ TeV_, _JHEP_ 06 (2018) 027, [1801.08427].
  * (19) S. Fajfer and D. Susič, _Colored scalar mediated nucleon decays to an invisible fermion_ , _Phys. Rev. D_ 103 (2021) 055012, [2010.08367].
  * (20) K. Fridell, J. Harz and C. Hati, _Probing baryogenesis with neutron-antineutron oscillations_ , _JHEP_ 11 (2021) 185, [2105.06487].
  * (21) S.-L. Chen and Y.-Q. Xiao, _The decomposition of neutron-antineutron oscillation operators_ , _JHEP_ 03 (2023) 065, [2211.02813].
  * (22) B. Fornal and B. Grinstein, _Dark Matter Interpretation of the Neutron Decay Anomaly_ , _Phys. Rev. Lett._ 120 (2018) 191801, [1801.01124].
  * (23) W. Tan, _Neutron Lifetime Anomaly and Mirror Matter Theory_ , _Universe_ 9 (2023) 180, [2302.07805].
  * (24) R. Allahverdi, B. Dutta and K. Sinha, _Baryogenesis and Late-Decaying Moduli_ , _Phys. Rev. D_ 82 (2010) 035004, [1005.2804].
  * (25) R. Allahverdi and B. Dutta, _Natural GeV Dark Matter and the Baryon-Dark Matter Coincidence Puzzle_ , _Phys. Rev. D_ 88 (2013) 023525, [1304.0711].
  * (26) R. Allahverdi, P. B. Dev and B. Dutta, _A simple testable model of baryon number violation: Baryogenesis, dark matter, neutron–antineutron oscillation and collider signals_ , _Phys. Lett. B_ 779 (2018) 262–268, [1712.02713].
  * (27) B. Dutta, Y. Gao and T. Kamon, _Probing Light Nonthermal Dark Matter at the LHC_ , _Phys. Rev. D_ 89 (2014) 096009, [1401.1825].
  * (28) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer et al., _The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations_ , _JHEP_ 07 (2014) 079, [1405.0301].
  * (29) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten et al., _An introduction to PYTHIA 8.2_ , _Computer Physics Communications_ 191 (2015) 159–177.
  * (30) DELPHES 3 Collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens et al., _DELPHES 3, A modular framework for fast simulation of a generic collider experiment_ , _JHEP_ 02 (2014) 057, [1307.6346].
  * (31) M. Cacciari, G. P. Salam and G. Soyez, _The anti- $k_{t}$ jet clustering algorithm_, _JHEP_ 04 (2008) 063, [0802.1189].
  * (32) CMS Collaboration, _Identification of b quark jets at the CMS Experiment in the LHC Run 2_ , _CMS Physics Analysis Summary CMS-PAS-BTV-15-001_ (2016) .
  * (33) CMS Collaboration, A. M. Sirunyan et al., _Identification of heavy-flavour jets with the CMS detector in pp collisions at 13 TeV_ , _JINST_ 13 (2018) P05011, [1712.07158].
  * (34) CMS Collaboration, V. Khachatryan et al., _Jet energy scale and resolution in the CMS experiment in pp collisions at 8 TeV_ , _JINST_ 12 (2017) P02014, [1607.03663].
  * (35) H. Voss, A. Hocker, J. Stelzer and F. Tegenfeldt, _TMVA, the Toolkit for Multivariate Data Analysis with ROOT_ , _PoS_ ACAT (2007) 040.
  * (36) L. Moneta, K. Belasco, K. S. Cranmer, S. Kreiss, A. Lazzaro, D. Piparo et al., _The RooStats Project_ , _PoS_ ACAT2010 (2010) 057, [1009.1003].