# SaiNet: Stereo aware inpainting behind objects with generative networks

Violeta Menéndez González1,2
<EMAIL_ADDRESS>Andrew Gilbert1
<EMAIL_ADDRESS>Graeme Phillipson2
<EMAIL_ADDRESS>Stephen Jolly2
<EMAIL_ADDRESS>Simon Hadfield1
<EMAIL_ADDRESS>1CVSSP, University of Surrey 2BBC R&D

###### Abstract

In this work, we present an end-to-end network for stereo-consistent image
inpainting with the objective of inpainting large missing regions behind
objects. The proposed model consists of an edge-guided UNet-like network using
Partial Convolutions. We enforce multi-view stereo consistency by introducing
a disparity loss. More importantly, we develop a training scheme where the
model is learned from realistic stereo masks representing object occlusions,
instead of the more common random masks. The technique is trained in a
supervised way. Our evaluation shows competitive results compared to previous
state-of-the-art techniques.

Figure 1: SaiNet: Inpainting behind objects using geometrically meaningful
masks.

##  1 Introduction

Image inpainting is the task of filling in the missing regions of an image
with perceptually plausible content. It has many vital applications in
computer vision and image processing: the removal of unwanted objects (e.g.
superimposed text), image and film restoration (e.g. scratches or cracks),
image completion (e.g. dis-occlusions), cinema post-production, among others.
Our work focuses on the under-explored problem of stereo-inpainting. This
lends itself to applications that need to see behind objects to generate
reasonable image fillers, for example in novel view synthesis of scenes,
object removal in stereoscopic video, 3D animation of still images, and dis-
occlusion in virtual reality environments.

This paper focuses on applications of inpainting which may improve view
synthesis in media production, this requires an approach that can take
advantage of multiple cameras, but doesn’t necessarily have the computer
capacity of other novel view approaches that reconstruct a whole 3D scene
representation. In addition, we want an approach that can generalise well to
unseen scenes and generate creative content without human input, therefore we
want to apply CNNs that can single-handedly propagate structures and textures
reasonably.

In previous works, traditional monocular techniques tried to achieve
inpainting by propagating local image structures and textures or copying
patches from the known areas of the image. This worked well for small or
narrow regions, but it was prone to generating visual inconsistencies in more
significant gaps. Early stereo techniques attempted to equivalently generate
consistent image output by mechanically warping the available data from the
other views [30], or completing the disparity images [19, 20], and then
proceeding similarly to the monocular inpainting approaches. However, in
recent years, Deep Learning (DL) techniques have taken advantage of large-
scale training data to create more semantically significant inpainting
outputs. Some works focused on learning embeddings of the images [22, 10],
while others developed different types of convolutional layers to be able to
handle more realistic irregular holes [15, 33]. However, the only DL
techniques that address the stereo inpainting problem [4, 17, 16] to date have
focused on artificial or unrealistic inpainting regions, or don’t enforce
multi-camera consistency.

In contrast, our approach focuses on inpainting one target image on
geometrically meaningful masks while using the information available from the
other viewpoint. Our network architecture is inspired by the 3D photography
generation work of Shih _et al_. [27] using a Partial Convolution [15]
architecture, which optimises the use of irregular masks at random locations.
Furthermore, we improve the inpainting task by adding colour edge information
following the idea by Nazeri _et al_. [21] in their work with EdgeConnect.

More importantly, we propose a novel stereo inpainting training mechanism.
Instead of using random image masks, which usually represent the physical
damage a picture can suffer, we use meaningful and geometrically-consistent
object masks that are not necessarily bounded within the image. We extend the
2D context/synthesis region approach proposed by [27] to use a bank of
geometrically-consistent 3D object masks. Ground-truth training examples are
generated from random virtual 3D objects placed at random locations in the
foreground of the scene, allowing us to have a fully self-supervised stereo
training approach. This data augmentation process addresses both the
significance of masked regions and the stereo data scarcity problem.
Furthermore, the resulting model is computationally efficient and able to
generalise to previously unknown scenes and occluding objects.

In summary, the contributions of this paper are:

  * •

A novel stereo-aware structure-guided inpainting model suitable for efficient
novel-view synthesis and free viewpoint VR applications.

  * •

First inpainting work to take full advantage of stereo-context with
geometrically-consistent object masks.

  * •

A novel stereo consistency loss attempting to ensure that inpainting results
are consistent with disoccluded information present in other views.

##  2 Background

#### Learnable inpainting

With the advancements of Deep Learning and the availability of large-scale
training data, deep Convolutional Neural Networks (CNNs) became a popular tool
for image prediction. Initial CNN models attempted to perform image inpainting
by using feature learning with Denoising Autoencoders [32], translation
variant interpolation [23], or exploiting the shape of the masks [14]. Yet all
these methods were only applicable to tiny and thin masks and lacked semantic
understanding of the scenes. With the addition of Generative Adversarial
Networks (GANs) [8], CNN architectures were able to extract meaningful
semantics from images and generate novel content. Pathak _et al_. [22] used an
encoder-decoder architecture to create a latent feature representation of the
image, which captured both the semantics and appearance of structures, but
struggled to maintain global consistency. Iizuka _et al_. [10] proposed using
both local and global context discriminators, which helped the local
consistency of generated patches and still held image coherence in the whole.
Yu _et al_. [34] added a contextual attention layer to aid the modelling of
long-term correlations between the distant information and the hole regions.

Traditional vanilla convolutions depend on the hole initialisation values,
which usually leads to visual artefacts. Liu _et al_. [15] proposed the use of
Partial Convolutions: masked and re-normalised convolutional filters
conditioned only on valid pixels. Yu _et al_. [33] extended this idea with
Gated Convolutions by generalising to a learnable dynamic features selection
mechanism. Previous works focused on centred rectangular holes, which may
cause methods to overfit to this kind of mask. Masked convolutions allowed
models to handle more realistic irregular holes. Liu _et al_. [15] studied the
effects when the holes are in contact with the image border and created a
large benchmark of irregular masks with varying sizes and locations. Many of
these methods still fail to reconstruct reasonable structures and usually
over-smooth surfaces. Some approaches [29, 21, 24] tackle this problem by
trying first to recover structural information to guide the inpainting of fine
details and textures. With a two-stage adversarial model, EdgeConnect [21]
first recovers colour edges, while StructureFlow [24] choose edge-preserved
smooth images as the global structure information.

#### Stereo Consistent Inpainting

There is little research done on stereoscopic image inpainting in the
framework of deep learning. Following a similar trajectory to monocular
approaches, traditional patch-based methods [19, 20, 30] find example patches
from the available parts of the image and fill the holes applying consistency
constraints. Wang _et al_. [30] simultaneously inpaint colour and depth images
using a greedy segmentation-based approach, inpainting first partial
occlusions using warping, and total occlusions with a depth-assisted texture
synthesis technique. Morse _et al_. [19] extend PatchMatch [1] to cross-image
searching and matching without explicit warping, using a completed disparity
map to guide the colour inpainting. Multi-view inpainting techniques such as
Gilbert _et al_. [7] create a dictionary of patches from multiple available
viewpoints that are then coherently selected and combined to recover the
missing region.

Figure 2: Model overview: Edge guidance, stereo context and disparity loss.

The first stereo inpainting approach using deep learning was made by Luo _et
al_. [16]. They use a double reprojection technique to generate image
occlusion masks from several new viewpoints, then apply Partial Convolutions
[15] to inpaint the holes, and aggregate the results in a layered depth image.
This technique shows good visual results on their own Keystone B&W dataset,
but they don’t take into account multi-view consistency and rely on depth maps
to reconstruct the image. Other techniques take advantage of both left and
right views, like Chen _et al_. [4]. They use an extension of Context Encoder
[22]. They inpaint left and right views simultaneously encoding both views and
aggregating them at the feature level. In addition, they introduced a local
consistency loss which helps preserve the inpainting consistency at a pixel
level. They applied this model to inpainting regular holes at the centre of
the image. Ma _et al_. [17] use a similar architecture for two different
tasks: reconstructing missing objects in one view that are available on the
other view and coherently inpainting the same holes in both views similar to
[4]. To do this, they use two different stereo consistency losses, a warping-
based consistency loss and a stereo-matching PSMNet-based [3] disparity-
reconstruction loss. However, because of a lack of ground-truth data for
object removal, they only train their model on corruption restoration data. In
contrast, our approach uses realistic and geometrically consistent foreground
object masks to explore inpainting behind objects in stereo scenes.

##  3 Approach

###  3.1 Model overview

An overall visualisation of our proposed model can be seen in Figure 2. It
consists of a deep neural network that follows a UNet-like architecture [25]
with partial convolutions [15]. The network takes the context and synthesis
areas of an object, where the context area is the background surrounding the
object. The synthesis area is the region behind the object (hole) that the
network will inpaint. In addition, the colour edges are fed into the network
for structural guidance. Finally, to enrich the inpainting and make the
network aware of the stereo view, a stereo-context image is added to the
input.

###  3.2 Object occlusion regions

An essential part of image inpainting specifies the type of missing regions
that the model needs to handle. Most previous approaches to inpainting have
focused on randomly shaped inpainting masks of limited complexity. This is
reasonable when dealing with image degradations such as scratches or removing
regions containing nuisance objects in 2D. However, for stereo inpainting
where we wish to maintain crisp object boundaries, this approach no longer
makes sense. We need inpainting masks that represent real image occlusions.
Therefore we propose a self-supervised approach where stereoscopic scenes are
augmented with geometrically valid inpainting masks, based on a virtual 3D
occluding object. This object is hallucinated in a stereo-consistent way over
both images, which allows us to collect “behind the object” ground-truth data.
As such, the network learns to fill in geometrically-meaningful holes with
background information, which can then be applied to actual object occlusions
in novel view synthesis applications.

Figure 3: Data generation process. A collection of context/synthesis regions
is created by extracting them from object boundaries in images on the COCO
dataset. Then they are randomly sampled, warped, and pasted onto different
images, forming the training dataset of ground truth context/synthesis regions

We generate these geometrically-valid masks from an unrelated dataset of
natural scenes with either ground truth depth or object segmentations. As
summarised in Algorithm 1, we first detect depth discontinuities [27] along
object boundaries and generate context and synthesis regions by propagating
this boundary towards the background image (context), and the foreground
object area (synthesis) (See Fig. 3 for a visualisation of these areas). This
way, we create a geometrically-meaningful bank of masks for inpainting.

Input: $\mathcal{N}=\\{\mathbf{I}:\mathbf{I}\text{ is a natural image}\\}$

Output: $\mathcal{M}=\\{(\mathbf{C}^{obj},\mathbf{S}^{obj})\mid\forall
obj\in\mathbf{I},\forall\mathbf{I}\in\mathcal{N}\\}$

for  _$\mathbf{I}$ in $\mathcal{N}$_ do

Find set of discontinuities $d_{\mathbf{I}}\equiv\\{d^{obj}_{\mathbf{I}}\mid
obj\text{ is an object in image }\mathbf{I}\\}$;

for  _$d^{obj}_{\mathbf{I}}$ in $d_{\mathbf{I}}$_ do

Propagate background around $d^{obj}_{\mathbf{I}}$ to generate context mask
$\mathbf{C}^{obj}$;

Propagate foreground around $d^{obj}_{\mathbf{I}}$ to generate synthesis mask
$\mathbf{S}^{obj}$;

end for

end for

Algorithm 1 Generation of geometrically-valid masks

These masks are varied and irregular, preventing our model overfitting one
type of mask. Furthermore, as opposed to most methods, our model doesn’t use
the whole image as context for the inpainting process, but just the region
closer to the object boundaries. Although this reduces the available
information that the network can learn from, it allows the network to narrow
its attention to the most relevant and meaningful area. However, this poses a
more challenging problem, as the context-to-synthesis area ratio is smaller,
and the masked regions are not necessarily bounded by context on all sides.

###  3.3 Stereo awareness

Our approach aims to make inpainting consistent across views in two different
ways. One is by enriching the network with extra available information. The
other is by enforcing a disparity loss on the output of the network (as
explained in Section 3.4). The main advantage of having two (or more) cameras
is the additional information we can extract to make the task of inpainting
“unknown” areas easier. For example, some colours or textures may be
completely occluded by the object in one view, but still be partially visible
from the other view. In this case, the additional input can provide strong
cues for the network to inpaint the occluded region. We make our system
stereo-aware by providing this extra information as input to the network by
warping the context mask of each object based on its estimated disparity value
into the additionally available view (See Algorithm 2). We use PSMNet [3] to
estimate this disparity and select the closest depth value to make sure the
object is situated at the front of all other objects in the scene. In other
words, we extract contextual information around the boundary of the occluding
object, in both views. Then we feed this extra context into the network to
learn to use it in filling in the synthesis area. In this way, we aid the
inpainting process by enriching the texture and colour information available.

Input: $\mathcal{D}=\\{(\mathbf{I}_{L},\mathbf{I}_{R}):\text{stereo pair of
images}\\}$

$\mathcal{M}=\\{(\mathbf{C}^{obj},\mathbf{S}^{obj})\mid\forall\text{ object
}obj\\}$

Output:
$\mathrm{Training\\_set}=\\{(\mathbf{CC}_{L},\mathbf{CS}_{L},\mathbf{E}_{L},\mathbf{CC}_{R})\mid\forall(\mathbf{I}_{L},\mathbf{I}_{R})\in\mathcal{D}\\}$

for  _$(\mathbf{I}_{L},\mathbf{I}_{R})$ in $\mathcal{D}$_ do

1\. Select random context and synthesis masks
$\mathbf{C}^{obj},\mathbf{S}^{obj}$ from the $\mathrm{Mask\\_Bank}$;

2\. Select a random position $x,y$ to situate the object at $\mathbf{I}_{L}$;

3\. Crop image $\mathbf{I}_{L}$ at $x,y$ with mask $\mathbf{C}^{obj}$ to
generate colour context region $\mathbf{CC}_{L}$;

4\. Crop image $\mathbf{I}_{L}$ at $x,y$ with mask $\mathbf{S}^{obj}$ to
generate colour synthesis region $\mathbf{CS}_{L}$;

5\. Generate edge map
$\mathbf{E}_{L}=\operatorname{Canny}(\mathbf{CC}_{L}+\mathbf{CS}_{L})$;

6\. Estimate depth map
$\mathbf{D}_{L}=\operatorname{PSMNet}(\mathbf{I}_{L},\mathbf{I}_{R})$;

7\. Crop image $\mathbf{D}_{L}$ at $x,y$ with mask $\mathbf{S}^{obj}$ to
generate depth synthesis region $\mathbf{DS}_{L}$;

8\. $disp=\max(\mathbf{DS}_{L})$;

9\. Reproject $\mathbf{C}^{obj}$ using $disp$ value onto $\mathbf{I}_{R}$ and
crop to generate stereo colour context region $\mathbf{CC}_{R}$;

end for

Algorithm 2 Stereo-aware training set generator

Several methods [21, 24, 29] have shown that structure-guided inpainting
performs better at reconstructing high frequency information accurately. Since
image structure is well-represented in its edge mask, superior results can be
obtained by conditioning an image inpainting network on edges in the missing
regions. For this reason, we feed the edge maps generated using Canny edge
detector [2] along with the colour information, as a bias to our network,
following a similar process to Nazeri _et al_. [21]. At test time, we estimate
the edges using a pre-trained EdgeConnect [21] model.

###  3.4 Stereo consistency Loss

Inspired by the work of Chen _et al_. [4], we propose a local consistency loss
which measures the consistency between the inpainted area in one view, and the
ground truth in the other view. In this way, we encourage the system to use
the stereo context; inpainting not just any perceptually acceptable
background, but specifically the one consistent with any partial observations.
The loss is illustrated in Fig. 2.

We compare a patch $P\left(i\right)$ around every pixel $i$ in the inpainted
area $\mathbf{S}\odot\mathbf{I}$ against a patch centred on the corresponding
pixel on the other view. $\mathbf{S}$ is the binary mask indicating the
synthesis region, $\mathbf{I}$ is the inpainted image, and $\odot$ denotes the
Hadamard product.

$\displaystyle\mathcal{L}_{disp}$
$\displaystyle=\frac{1}{\left|\mathbf{S}\right|}\sum_{i\in\mathbf{S}\odot\mathbf{I}}\overleftarrow{cost}\left(i\right),$
(1) $\displaystyle\overleftarrow{cost}\left(i\right)$
$\displaystyle=1-\Phi\left(P\left(i\right),P\left(\overleftarrow{W}\left(i\right)\right)\right)$
(2)

where $\overleftarrow{W}$ is the warping function corresponding to a change
from source to target view, using the disparity estimated by PSMNet [3]. We
use a Normalised Cross-Correlation (NCC) as our stereo matching cost ($\Phi$)
which works well with back-propagation.

$\Phi(X,Y)=\dfrac{\left\|X\odot
Y\right\|_{1,1}}{\left\|X\right\|_{F}\left\|Y\right\|_{F}}$ (3)

here $\left\|\cdot\right\|_{1,1}$ and $\left\|\cdot\right\|_{F}$ are the
1-entrywise and Frobenious matrix norms respectively.

###  3.5 Inpainting losses

In addition to the disparity loss, other per-pixel similarity losses and
losses based on deep features are used to enforce perceptually realistic
results. First, two per-pixel reconstruction losses are defined over the
synthesis and context regions, these losses help guiding the inpainting of the
missing areas, as well as making sure that context and synthesis areas are
recovered consistently and with smooth boundaries.

$\displaystyle\mathcal{L}_{synthesis}$
$\displaystyle=\frac{1}{N_{\mathbf{I}_{gt}}}\left\|\mathbf{S}\odot\left(\mathbf{I}-\mathbf{I}_{gt}\right)\right\|_{1},$
(4) $\displaystyle\mathcal{L}_{context}$
$\displaystyle=\frac{1}{N_{\mathbf{I}_{gt}}}\left\|\mathbf{C}\odot\left(\mathbf{I}-\mathbf{I}_{gt}\right)\right\|_{1}$
(5)

where $\mathbf{S}$ and $\mathbf{C}$ are the binary masks indicating synthesis
and context regions respectively, $N_{\mathbf{I}_{gt}}$ is the total number of
pixels, $\mathbf{I}$ is the inpainted result, and $\mathbf{I}_{gt}$ is the
ground truth image. In addition, we include two deep feature losses from
Johnson _et al_. [12], based on VGG-16 [28] embeddings, that measure high-
level perceptual and semantic differences. Firstly

$\displaystyle\mathcal{L}_{perceptual}=\sum_{p=0}^{P-1}\frac{\left\|\Psi_{p}\left(\mathbf{I}\right)-\Psi_{p}\left(\mathbf{I}_{gt}\right)\right\|_{1}}{N_{\Psi_{p}}}$
(6)

where, $\Psi_{p}(\cdot)$ is the output of the p’th layer from VGG-16 [28], and
$N_{\Psi_{p}}$ is the total number of elements in the layer. Secondly, the
style loss is defined as,

$\displaystyle\mathcal{L}_{style}=\sum_{p=0}^{P-1}\frac{1}{C_{p}C_{p}}\left\|K_{p}\left[\left(\Psi^{\mathbf{I}}_{p}\right)^{\intercal}\Psi^{\mathbf{I}}_{p}-\left(\Psi^{\mathbf{I}_{gt}}_{p}\right)^{\intercal}\Psi^{\mathbf{I}_{gt}}_{p}\right]\right\|_{1}$
(7)

where $K_{p}=\frac{1}{C_{p}H_{p}W_{p}}$ is a normalisation factor, and
$C_{p},H_{p},W_{p}$ are the number of channels, height, and width of the
output $\Psi_{p}(\cdot)$.

These perceptual losses encourage the network to create images with similar
content and similar feature representations. The style loss ensures that the
style of the output images resemble the input in colour, textures, etc.
Finally, a total variation loss is used as a smooth regularization.

$\displaystyle\mathcal{L}_{tv}$
$\displaystyle=\sum_{\left(i,j\right)\in\mathbf{S}}\frac{\left\|\mathbf{I}(i,j+1)-\mathbf{I}(i,j)\right\|_{1}}{N_{\mathbf{I}_{gt}}}$
(8)
$\displaystyle+\sum_{\left(i,j\right)\in\mathbf{S}}\frac{\left\|\mathbf{I}(i+1,j)-\mathbf{I}(i,j)\right\|_{1}}{N_{\mathbf{I}_{gt}}}$
(9)

where the $\mathbf{S}$ denotes the pixels in the synthesis region.

Figure 4: Real dataset evaluation over Middlebury [11] using Canny edge
detector. The zoomed-in crop of yellow area is visualised as “Ground Truth”.

Similar to Liu _et al_. [15], we use the following weights to combine all
these losses to yield the final training objective: $\lambda_{synthesis}=6$,
$\lambda_{context}=1$, $\lambda_{perceptual}=0.05$, $\lambda_{style}=120$,
$\lambda_{tv}=0.1$, $\lambda_{disparity}=0.1$. The same parameters are used
for all evaluations.

###  3.6 Datasets

Good quality, natural stereo datasets are very hard to come by. This is a
problem for training deep neural networks, which usually require a high number
of images to extract meaningful statistical information. Our approach to data
collection intrinsically performs data augmentation, as the random sampling of
context-synthesis areas makes it possible to use different samplings of the
same image without overfitting.

For training we have used three different datasets: SceneFlow [18]:
FlyingThings3D, Driving, and Middlebury [26]. FlyingThings3D consists of
21,818 frames from 2,247 scenes, containing everyday objects flying around in
a randomised way. This is ideal for training CNNs due to the large amount of
data and variety of objects. Driving is a more naturalistic-looking dynamic
street scene resembling the KITTI dataset [6]. It contains 4400 images from
one scene. On the other hand, the Middlebury dataset consists of only 33 pairs
of stereo images of natural scenes. Even though this dataset is not big enough
to train a Deep Learning model, we are able to perform transfer learning and
generate pleasant results over real world data (See Fig. 4).

These datasets contain ground truth disparity maps, but for our model we have
included a disparity estimation step using PSMNet so we don’t rely on existing
ground truth data. This makes it fairer to compare to other models that use a
similar approach, as well as being more relevant to our application to media
production, where we may have several views from the same scene, but no depth
information.

Figure 5: Qualitative inpainting results for FlyingThings3D. Baseline is Shi _et al_. [27]. The zoomed-in crop of the yellow area is visualised in the “Ground Truth” column. Table 1: Quantitative results. Image quality & stereo consistency of different models. Bold is best. $\star$ values are from their paper. Dataset | Model | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | DispE (%)$\downarrow$
---|---|---|---|---|---
FlyingThings3D | Shih _et al_. [27] | 28.32 | 0.8589 | 0.0707 | 7.96
Ours | 30.50 | 0.8643 | 0.0556 | 7.67
Driving | Shih _et al_. [27] | 30.46 | 0.969 | 0.1141 | 9.94
Chen _et al_. [4]⋆ | 22.38 | 0.959 | - | 7.79
Ma _et al_. [17]⋆ | 23.20 | 0.964 | - | 4.72
| Ours | 34.94 | 0.977 | 0.0628 | 8.01
Table 2: Ablation study. Compare the accuracy of different stages of the model over all regions. ‘Baseline’ is the monocular inpainting model, ‘Stereo’ is the model + stereo context, ‘Disp’ is the model + disparity loss, and ‘Full’ if the model with both stereo context and disparity loss. Bold is best result. Blue are results in synthesis regions only. Model | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | DispE (%)$\downarrow$
---|---|---|---|---
Baseline | 28.32 (22.26) | 0.8589 (0.5619) | 0.0707 (0.0676) | 7.96
Ours (Stereo) | 29.41 (23.61) | 0.8604 (0.5597) | 0.0625 (0.0582) | 7.68
Ours (Disp) | 29.79 (24.00) | 0.8619 (0.5684) | 0.0570 (0.0569) | 7.71
Ours (Full) |  30.50 (24.70) |  0.8643 (0.5771) |  0.0556 (0.0539) | 7.67

###  3.7 Experiment setup

The network is trained using a batch size of 8 and $256\times 256$ images. The
model is optimised using Adam optimiser [13] and a learning rate of 0.1. A
model has been trained for each different dataset. As FlyingThings3D is 3 to 5
times bigger than the other datasets, a transfer learning approach has been
followed where the model is trained on FlyingThings3D first and then fine-
tuned over Driving, and Middlebury.

For fair comparison to the results of Chen _et al_. [4] and Ma _et al_. [17],
we have trained our Driving model using $128\times 128$ square context masks
and $64\times 64$ centred synthesis masks. Our baseline model is Shih _et al_.
[27] 3D photography colour inpainting network, which has been trained in the
same fashion as our model, and conditioned over depth edges instead of colour
as per their original pipeline.

For training, we generate edge maps using Canny [2] edge detector following
EdgeConnect [21] approach. At test time, we apply pre-trained EdgeConnect
models to generate the synthesis area edges, using the pre-trained model over
Places2 [36] for our FlyingThings3D, and Middlebury, and a pre-trained model
over Paris StreetView [5] for our Driving.

##  4 Results and Discussion

In this section we show different evaluations and comparatives that
demonstrate the value of our work. We train our model on three different
datasets as explained in Section 3.7, and we compare its accuracy and
consistency against state-of-the-art methods. We also perform an ablation
study to evidence the benefits of the different contributions of our model.

###  4.1 Evaluation of Accuracy

There is no perfect numerical metric to evaluate image inpainting outputs
given the variety of possible valid results. For the purpose of quantifying
how well our model performs, we make use of several popular metrics that
measure different characteristics of an image. To measure image quality, we
use Peak Signal-To-Noise Ratio (PSNR) [9] and Structural SIMilarity (SSIM)
[31] index. PSNR shows the overall pixel consistency, while SSIM measures the
coherence of local structures. These metrics assume pixel-wise independence,
which may assign favourable scores to perceptually inaccurate results. For
this reason, we also include the use of a Learned Perceptual Image Patch
Similarity (LPIPS) [35] metric, which aims to capture human perception using
deep features.

The stereo consistency is quantified using the disparity error metric from
[17], which counts the erroneous pixels of the PSMNet estimated disparity map
of the inpainted image, compared against the ground truth111The definition of
[17] has a typo where the absolute error $\left|d^{i}_{est}-d^{i}_{gt}\right|$
is replaced by $d^{i}_{est}$.. Given the inpainted image $\mathbf{I}$, for
every pixel $i$ we consider its estimated disparity $d^{i}_{est}$ to be
erroneous iff the absolute error against the equivalent pixel in the ground
truth disparity image $d^{i}_{gt}$ is greater than $p_{1}$ and its relative
error greater than $p_{2}$ (we use $p_{1}=3$ and $p_{2}=0.05$). This is
described in equation 10, where $N$ is the total number of pixels, and $[\;]$
is the Iverson bracket.

$\displaystyle\mathit{DispE}=\frac{1}{N}\sum_{i\in\mathbf{I}}\Biggl{[}\Bigl{(}\left|d^{i}_{est}-d^{i}_{gt}\right|$
$\displaystyle>p_{1}\Bigr{)}$ (10)
$\displaystyle\And\Biggl{(}\frac{\left|d^{i}_{est}-d^{i}_{gt}\right|}{d^{i}_{gt}}$
$\displaystyle>p_{2}\Biggr{)}\Biggr{]}$ (11)

###  4.2 Inpainting comparison

We perform a quantitative comparison of our inpainting model against other
state-of-the-art methods [27, 4, 17], following the experiment setup described
in Section 3.7. Results can be seen in Table 1.

We can observe our model performs better across all metrics compared with the
baseline model of Shih _et al_. [27]. Our model also performs competitively
against other stereo inpainting models [4, 17], showing a superior image
inpainting quality with an improvement on PSNR values of 50%, and some
improvement to SSIM. Due to the nature of our mask generation process, our
stereo context information is quite narrow, limiting the visible area that our
network can learn from. Despite this, our model accomplishes similar results
to the stereo consistency of Chen _et al_. [4]. The image quality of the
inpainting is superior on the Driving dataset, which was trained using square
centred masks to match the experimental setup of [4, 17]. Meanwhile, the
object-like occlusion masks used on the FlyingThings3D dataset, which are not
fully bounded, are much more challenging.

A qualitative example is shown in Figure 5222For further results and analysis
we refer the readers to the supplementary material. Despite having access to
depth edges, Shi _et al_. struggles to produce sharp object boundaries in the
inpainted region. Meanwhile SaiNet is able to use stereo context to inpaint
sharp boundaries using colour edge information. This is evidenced by Shih _et
al_. success recovering the green bar in the first example, but failing on the
colour edge of the second example. However, as shown in the 4th example, our
technique still struggles to inpaint especially intricate structures which are
not visible through stereo context. Nevertheless it produces sharper and more
visually pleasing results.

###  4.3 Ablation Study

In the interest of proving the contribution of every stage to the accuracy of
the model, we have studied its performance removing the key contributions.
Results presented in Table 2 show that every part of the model performs better
than the baseline, with the combination of all modules having the best
performance across all metrics. It is interesting to note that the use of a
disparity loss provides the largest individual benefit in terms of stereo
consistency.

##  5 Conclusion

We introduced a new stereo-aware learned inpainting model that enforces stereo
consistency on its output, trained in a self-supervision fashion over
geometrically meaningful masks representing object occlusions. This technique
improved over state-of-the-art models by up to 50% PSNR, and we demonstrated
its performance over several diverse datasets. As future work, it would be
helpful to explore how we could extend similar techniques to cope with the
challenges that wide-baseline non-parallel cameras would provide.

#### Acknowledgements.

This work was partially supported by the British Broadcasting Corporation
(BBC) and the Engineering and Physical Sciences Research Council’s (EPSRC)
industrial CASE project “Generating virtual camera views with generative
networks” (voucher number 19000033).

## References

  * [1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing.  ACM Transactions on Graphics, Aug. 2009.
  * [2] John Canny.  A Computational Approach to Edge Detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, Nov. 1986.
  * [3] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid Stereo Matching Network.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018.
  * [4] Shen Chen, Wei Ma, and Yue Qin.  CNN-Based Stereoscopic Image Inpainting.  In Int. Conf. on Image and Graphics (ICIG), Nov. 2019.
  * [5] Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei A. Efros.  What Makes Paris Look Like Paris?  ACM Transactions on Graphics (SIGGRAPH), 2012.
  * [6] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? The KITTI vision benchmark suite.  In IEEE Conference on Computer Vision and Pattern Recognition, June 2012.
  * [7] Andrew Gilbert, Matt Trumble, Adrian Hilton, and John Collomosse.  Inpainting of Wide-Baseline Multiple Viewpoint Video.  IEEE Transactions on Visualization and Computer Graphics, Dec. 2018\.
  * [8] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, Dec. 2014.
  * [9] Q. Huynh-Thu and M. Ghanbari.  Scope of validity of PSNR in image/video quality assessment.  Electronics Letters, 2008.
  * [10] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and Locally Consistent Image Completion.  ACM Transactions on Graphics, 2017.
  * [11] R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs.  Large Scale Multi-view Stereopsis Evaluation.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 2014.
  * [12] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution.  In European Conference on Computer Vision, Oct. 2016.
  * [13] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  International Conference on Learning Representations, 2014.
  * [14] Rolf Köhler, Christian Schuler, Bernhard Schölkopf, and Stefan Harmeling.  Mask-Specific Inpainting with Deep Neural Networks.  In Pattern Recognition - 36th German Conference, GCPR. Springer International Publishing, 2014.
  * [15] Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image Inpainting for Irregular Holes Using Partial Convolutions.  Proceedings of the European Conference on Computer Vision (ECCV), Apr. 2018.
  * [16] Xuan Luo, Yanmeng Kong, Jason Lawrence, Ricardo Martin-Brualla, and Steven M. Seitz.  KeystoneDepth: History in 3D.  In 2020 International Conference on 3D Vision (3DV), Nov. 2020.
  * [17] Wei Ma, Mana Zheng, Wenguang Ma, Shibiao Xu, and Xiaopeng Zhang.  Learning across views for stereo image completion.  IET Computer Vision, 2020.
  * [18] Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [19] B. Morse, J. Howard, S. Cohen, and B. Price.  PatchMatch-Based Content Completion of Stereo Image Pairs.  In 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), Oct. 2012.
  * [20] Tai-Jiang Mu, Ju-Hong Wang, Song-Pei Du, and Shi-Min Hu.  Stereoscopic image completion and depth recovery.  The Visual Computer: International Journal of Computer Graphics, June 2014.
  * [21] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi.  EdgeConnect: Structure Guided Image Inpainting using Edge Prediction.  In IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Oct. 2019.
  * [22] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros.  Context Encoders: Feature Learning by Inpainting.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Apr. 2016.
  * [23] Jimmy SJ. Ren, Li Xu, Qiong Yan, and Wenxiu Sun.  Shepard convolutional neural networks.  In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, Dec. 2015.
  * [24] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, and Ge Li.  StructureFlow: Image Inpainting via Structure-Aware Appearance Flow.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In Medical Image Computing and Computer-Assisted Intervention – MICCAI, 2015.
  * [26] D. Scharstein, H. Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nesic, Xi Wang, and P. Westling.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth.  In GCPR, 2014.
  * [27] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3D Photography using Context-aware Layered Depth Inpainting.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 2020.
  * [28] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  International Conference on Learning Representations, Apr. 2015\.
  * [29] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C. C. Jay Kuo.  SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting.  British Machine Vision Conference (BMVC), May 2018.
  * [30] L. Wang, H. Jin, R. Yang, and M. Gong.  Stereoscopic inpainting: Joint color and depth completion from stereo images.  In IEEE Conference on Computer Vision and Pattern Recognition, June 2008.
  * [31] Wang, Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: From error visibility to structural similarity.  IEEE Transactions on Image Processing, Apr. 2004.
  * [32] Junyuan Xie, Linli Xu, and Enhong Chen.  Image Denoising and Inpainting with Deep Neural Networks.  Advances in Neural Information Processing Systems, Jan. 2012.
  * [33] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang.  Free-Form Image Inpainting With Gated Convolution.  In IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 2019.
  * [34] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Generative Image Inpainting with Contextual Attention.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jan. 2018.
  * [35] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018.
  * [36] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 Million Image Database for Scene Recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, June 2018.


11institutetext:  Istituto Nazionale di Fisica Nucleare (INFN), Sezione di
Firenze, Italy 22institutetext: Department of Information Engineering (DINFO),
University of Firenze, Italy 33institutetext: Istituto Nazionale di Fisica
Nucleare (INFN), Sezione di Milano-Bicocca, Italy 44institutetext: Department
of Physics, University of Milano-Bicocca, Italy 55institutetext: European
Organization for Nuclear Research (CERN), Switzerland 66institutetext:
Department of Physics and Astronomy, University of Manchester, United Kingdom
77institutetext: Affiliated with an institute covered by a cooperation
agreement with CERN 88institutetext: Department of Physics, University of
Ferrara, Italy 99institutetext: Laboratoire de Physique de Clermont (LPC),
UniversitÃ Clermont Auvergne, France

# The LHCb ultra-fast simulation option, Lamarr

design and validation

Lucio Anderlini 11 Matteo Barbetti 1122<EMAIL_ADDRESS>Simone Capelli
3344 Gloria Corti 55 Adam Davis 66 Denis Derkach 77 Nikita Kazeev 77 Artem
Maevskiy 77 Maurizio Martinelli 3344 Sergei Mokonenko 77 Benedetto G. Siddi 88
Zehua Xu on behalf of the LHCb Simulation Project 99

###### Abstract

Detailed detector simulation is the major consumer of CPU resources at LHCb,
having used more than 90% of the total computing budget during Run 2 of the
Large Hadron Collider at CERN. As data is collected by the upgraded LHCb
detector during Run 3 of the LHC, larger requests for simulated data samples
are necessary, and will far exceed the pledged resources of the experiment,
even with existing fast simulation options. An evolution of technologies and
techniques to produce simulated samples is mandatory to meet the upcoming
needs of analysis to interpret signal versus background and measure
efficiencies. In this context, we propose Lamarr, a Gaudi-based framework
designed to offer the fastest solution for the simulation of the LHCb
detector. Lamarr consists of a pipeline of modules parameterizing both the
detector response and the reconstruction algorithms of the LHCb experiment.
Most of the parameterizations are made of Deep Generative Models and Gradient
Boosted Decision Trees trained on simulated samples or alternatively, where
possible, on real data. Embedding Lamarr in the general LHCb Gauss Simulation
framework allows combining its execution with any of the available generators
in a seamless way. Lamarr has been validated by comparing key reconstructed
quantities with Detailed Simulation. Good agreement of the simulated
distributions is obtained with two-order-of-magnitude speed-up of the
simulation phase.

##  1 Introduction

The LHCb experiment LHCb:2008vvz  has been originally designed to study rare
decays of particles containing $b$ and $c$ quarks produced at the Large Hadron
Collider (LHC). The LHCb detector is a single-arm forward spectrometer
covering the pseudorapidity range of $2<\eta<5$, that includes a Tracking
system and a Particle Identification (PID) system LHCb:2014set . The Tracking
system provides high-precision measurements of the momentum $p$ of charged
particles and the position of primary vertices. Different types of charged
hadrons are separated using the response of two ring-imaging Cherenkov (RICH)
detectors. Photons, electrons and hadrons are identified by the calorimeter
system relying on an electromagnetic calorimeter (ECAL) and a hadron
calorimeter (HCAL). Finally, a dedicated system named MUON identifies muons
alternating layers of iron and multi-wire proportional chambers. The RICH,
calorimeters and MUON detectors are part of the PID system.

Interpreting signal, rejecting background contributions and performing
efficiency studies requires to have a full understanding of its data sample,
from the high-energy collisions to the set of physics processes responsible
for the detector high-level response. This kind of studies greatly benefits
from the use of simulated samples. At LHCb, the simulation production mainly
relies on the Gauss framework Clemencic:2011zza  that implements the
generation and simulation phases, and is based on the Gaudi processing
framework Barrand:2001ny . The high-energy collisions and all the physics
processes that produce the set of particles (e.g., muons, pions, kaons or
protons) able to traverse the LHCb spectrometer are simulated during the
_generation phase_ using software like Pythia8 Sjostrand:2007gs  and EvtGen
Lange:2001uf . The radiation-matter interactions between the detector
materials and the traversing particles are reproduced during the _simulation
phase_ that aims to compute the energy deposited in the active volumes and
relies on the Geant4 toolkit Allison:2006ve . Then, a separate application
converts the energy deposits into raw data compatible with the real one
collected by LHCb.

The simulation of all the physics events occurring within the detector is the
major consumer of CPU resources at LHCb, having used more than 90% of the
total computing budget during LHC $\mathrm{Run\leavevmode\nobreak\ 2}$. The
upgraded version of the experiment is designed to collect one-order-of-
magnitude larger data samples during $\mathrm{Run\leavevmode\nobreak\ 3}$.
Meeting the upcoming and future requests for simulated samples is not
sustainable relying only on the traditional _detailed simulation_. For this
reason, the LHCb Collaboration is spending great efforts in modernizing the
simulation software stack through the novel experiment-independent framework
Gaussino 111Visit https://gaussino.docs.cern.ch for additional details.
Mazurek:2021abc ; Mazurek:2022tlu  on which a newer version of Gauss will be
built on, and in developing faster simulation options, some of which also
powered by machine learning algorithms Chekalina:2018hxi ; Maevskiy:2019vwj ;
Anderlini:2022ofl ; Barbetti:2023bvi .

##  2 Fast simulation VS. ultra-fast simulation

Simulating all the physics processes of interest for LHCb is extremely
expensive in terms of computing resources, especially the Geant4-based step
that is the major CPU-consumer. Speeding up the computation of the energy
deposits or, more generally, the detector response is mandatory to satisfy the
demand for simulations expected for $\mathrm{Run\leavevmode\nobreak\ 3}$ and
those that will follow. Actually, this is a shared problem across the High
Energy Physics (HEP) community that is collectively facing it, including by
exploiting the latest achievements in Computer Science and adapting _deep
generative models_ to parameterize the low-level response of the various
experiments Paganini:2017dwg ; Krause:2021ilc ; Amram:2023onf . The literature
refers to this kind of strategies with the term _fast simulation_. Fast
simulations share their data processing scheme and the reconstruction step
with the detailed simulation (as depicted in Figure 1), and are proven capable
of reducing the computation cost of a simulated sample up to a factor of 20.

Figure 1:  Schematic representation of the data processing flow in the
_detailed_ (top), _fast_ (center) and _ultra-fast_ (bottom) simulation
paradigms.

To meet the upcoming and future requests for simulated samples, the LHCb
Collaboration is also considering a more radical approach based on the so-
called _ultra-fast simulation_ paradigm. In this case, the aim is to directly
reproduce the high-level response of the detector relying on a set of
parameterizations developed to transform generator-level particles information
into reconstructed physics objects as schematically represented in Figure 1
(bottom). Such parameterizations can still be built using generative models,
like _Generative Adversarial Networks_ (GAN), proven to succeed in reproducing
the high-level response of the LHCb detector Ratnikov:2023wof  and offering
reliable synthetic simulated samples Anderlini:2022ckd . Following pioneering
studies on the ultra-fast simulation of the electromagnetic calorimeter based
on GANs Musella:2018rdi , the CMS Collaboration has recently started
developing a full-scope ultra-fast simulation based on _Normalizing Flow_ ,
named FlashSim Vaselli:2858890 .

##  3 Lamarr: the LHCb ultra-fast simulation framework

Lamarr Anderlini:2022ofl ; Barbetti:2023bvi  is the official ultra-fast
simulation framework for LHCb, able to offer the fastest options for
simulation. Originating from the attempt of an LHCb customized version of
Delphes deFavereau:2013fsa ; Siddi:2019abc , Lamarr is an independent project
retaining only the inspiration of its modular layout from Delphes. In
particular, the Lamarr framework consists of a pipeline of modular
parameterizations, most of which based on machine learning algorithms,
designed to take as input the particles generated by the physics generators
and provide as output the high-level response of the various LHCb sub-
detectors.

The Lamarr pipeline can be logically split in two separated chains according
to the charge of the generated particles. We expect that charged particles
leave a mark in the Tracking system that Lamarr characterizes in terms of
acceptance, efficiency and resolution as described in Section 3.1. The
reconstructed tracking variables are then used to compute the response of the
PID system for a set of traversing charged particles (muons, pions, kaons or
protons) as detailed in Section 3.2. In case of neutral particles (e.g.,
photons), the calorimeters play a key role and, since multiple photons can
concur to the energy of a single calorimetric cluster, parameterizing
particle-to-particle correlation effects is of major relevance. The solutions
under investigation are reported in Section 3.3. The Lamarr pipelines
described above are shown in Figure 2.

###  3.1 Tracking system

One of the aims of the LHCb Tracking system is to measure the momentum $p$ of
charged particles (i.e., electrons, muons, pions, kaons and protons),
exploiting the deflection of their trajectories due to the dipole magnet
located in between the tracking detectors. Hence, the first step of the
_charged chain_ reported in Figure 2 is the propagation through the magnetic
field of the particles provided by the physics generators. Lamarr
parameterizes the particle trajectories as two rectilinear segments with a
single deflection point (inversely proportional to the transverse momentum
$p_{T}$), implementing the so-called _single $p_{T}$ kick_ approximation.

Figure 2:  Scheme of the Lamarr modular pipeline. According to the charge of
the particle provided by the physics generator, two sets of parameterizations
are defined: the charged particles are passed through the Tracking and PID
models, while the neutral ones follow a different path where the calorimeter
modeling plays a key role.

The next step requires to select the subset of tracks that fall within the
LHCb geometrical acceptance and that have any chance to be reconstructed. To
this end, Lamarr uses _Gradient Boosted Decision Trees_ (GBDT) trained to
learn the fraction of candidates that are in the acceptance as a function of
the kinematic information provided by the physics generators. Given a
generated track in acceptance, we ask whether the latter will be reconstructed
and, in case of positive answer, which tracking detectors are involved in the
reconstruction procedure. Lamarr statistically infers such information, namely
the tracking efficiency, relying on _neural networks_ trained to perform a
multi-class classification according to the track kinematics. A major effort
is ongoing to improve the performance of the efficiency model on the basis of
the type of tracks and particle species (i.e., electrons, muons or hadrons).

At this point, Lamarr disposes of the subset of the generated particles that
can be considered as reconstructed tracks, but their kinematics and geometry
are still identical to those provided by the physics generators. The smearing
of these features, mimicking the effect of the reconstruction, is achieved
using GANs. Driven by a _binary cross-entropy_ loss function and powered by
_skip connections_ , GANs succeed in describing the resolution effects due to,
for example, multiple scattering phenomena, only relying on track kinematic
information at generator-level as input conditions. A similar GAN-based
architecture is used to provide the correlation matrix obtained from the
Kalman filter adopted in the reconstruction algorithm to define the position,
slope and curvature of each track.

Stacking the parameterizations described above, Lamarr is able to provide the
high-level response of the LHCb Tracking system. The resulting reconstructed
quantities can be further processed using the LHCb analysis software to
combine the parameterized tracks into decay candidates as depicted by the
green slot in Figure 1 (bottom).

###  3.2 Particle identification system

To accomplish the LHCb physics program, disposing of a high-performance PID
system is crucial since it allows for discriminating the various particle
species that traverse the detector. Lamarr provides parameterizations for the
majority of the charged particles for which the PID detectors are relevant
(i.e., muons, pions, kaons or protons). Specialized parameterizations for the
electrons, encoding the multiple scattering and Bremsstrahlung emission
contributions in the interaction with the detector materials, is planned as
future development.

Identifying these subset particles involves mainly the RICH and MUON
detectors, while the role played by the calorimeters is minor. In general, we
expect that the response of the PID system depends only on the specie of the
traversing particle, its kinematics, and the detector occupancy. According to
these dependencies, Lamarr provides the high-level response for both the
detectors using GAN-based models properly conditioned Maevskiy:2019vwj ;
Anderlini:2022ckd . Given the particle specie from the physics generators, its
kinematic information results from the Lamarr Tracking modules, while the
detector occupancy is described by the total number of tracks traversing the
detector.

In real data, the combination of the responses from RICH detectors,
calorimeters, MUON system and a binary muon-identification criterion
implemented via FPGA and named isMuon allows to compute the higher-level
response of the PID system, referred to as GlobalPID variables. The
parameterization of the GlobalPID variables still relies on conditioned GANs,
adding as input what results from the RichGAN and MuonGAN models. The binary
output of a neural-network-based implementation of isMuon is used as
additional input feature, while no explicit calorimeters contribution is
defined leaving the missing information problem to the generator _latent
space_.

GAN-based models, driven by a _Wasserstein distance_ loss function and trained
using a Lipschitz-constrained discriminator Terjek:2020 , succeed in
describing the high-level response of the RICH and MUON systems. Chaining
together different GANs, Lamarr is also able to provide the higher-level
response of the LHCb PID system, injecting an implicit contribution from the
calorimeters.

###  3.3 Electromagnetic calorimeter

Providing a parameterization for the electrons requires describing the
response to Bremsstrahlung photons by the LHCb ECAL detector. Since interested
by a multitude of secondary particles, the detailed simulation of the
calorimeter system is the most computationally expensive step in the
simulation pipeline. The latter is a shared problem across the HEP community,
that is investing great efforts in tuning deep generative models to properly
parameterize the energy deposited in the calorimeter cells Chekalina:2018hxi ;
Paganini:2017dwg ; Krause:2021ilc ; Amram:2023onf . Such studies belong to the
fast-simulation paradigm that aims to reduce the Geant4 use, providing models
for the low-level response of the various experiments.

The current version of Lamarr provides a simplified parameterization for the
LHCb calorimeter, designed for detector studies and based on a fast-simulation
approach. Disposing information at the calorimeter cell level requires running
reconstruction algorithms to obtain analysis-level quantities that may become
rather CPU-expensive for high-multiplicity events. In addition, since non-
physical strategies are used to simulate the energy deposits (as is the case
for GANs), there is no certainty that the reconstruction software stack can
correctly reproduce the expected distributions for the high-level variables
Rogachev:2022hjg . Hence, the Lamarr project is actively working to provide an
ultra-fast solution for the ECAL detector.

Reproducing the calorimeter high-level response is a non-trivial task since
traditional generative models rely on the hypothesis that an unambiguous
relation between the generated particle and the reconstructed object
exists222To a first approximation, the response of the Tracking and PID
systems satisfy this condition.. Instead, the presence of merged $\pi^{0}$ and
Bremsstrahlung photons may lead to having $n$ generated particles responsible
for $m$ reconstructed objects (in general with $n\neq m$). A strategy to face
this particle-to-particle correlation problem can be built using techniques
designed in the context of Language Modeling, describing the calorimeter
simulation as a _translation problem_. To this end, _Graph Neural Network_
(GNN) Scarselli:2009abc  and _Transformer_ Vaswani:2017abc  models are
currently under investigation.

Both the models are designed to process a sequence of $n$ generated photons
and infer the kinematics of a sequence of $m$ reconstructed clusters. The non-
trivial correlations between any particles of the source sequence (photons)
and the target one (clusters) rely on the _attention mechanism_
Vaswani:2017abc ; Brody:2022abc . To improve the quality of the resulting
parameterizations, the training of both GNN and Transformer-based models is
driven by an adversarial procedure (similarly to what occurs for GANs). The
discriminator is currently implemented through a _Deep Sets_ model
Zaheer:2017abc , while further studies are ongoing to replace it with a second
Transformer Lee:2022abc . Considering the complexity of the problem, the
preliminary results are promising as depicted in Figure 3, where the joint
action of Transformer and Deep Sets succeeds in deriving the energy
distribution on the ECAL face. The center of the calorimeter has not active
material since is used to host the LHC beam pipe. It should be pointed out
that no constraints are applied to the model output to reproduce such
conditions, and that the empty space shown in Figure 3 (right) is the result
of the adversarial training procedure.

Figure 3:  Distribution of the $(x,y)$-position of the reconstructed clusters
on the LHCb ECAL face for a $2000\times 1500\leavevmode\nobreak\ \rm{mm}^{2}$
frame placed around the center. The geometrical information is combined with
the energy signature properly weighting each bin entry. What obtained from
detailed simulation is reported on the left, while the predictions of an
adversarial trained Transformer model is shown on the right. The corresponding
LHCB-FIGURE is in preparation.

##  4 Validation campaign and timing performance

The ultra-fast philosophy at the base of the Lamarr framework is being
validated by comparing the distributions obtained from machine-learnt models
trained on detailed simulation and the ones resulting from standard simulation
strategies. In particular, we will briefly discuss the validation studies
performed for the charged particles pipeline using simulated
$\Lambda_{b}^{0}\to\Lambda_{c}^{+}\mu^{-}\bar{\nu}_{\mu}$ decays with
$\Lambda_{c}^{+}\to pK^{-}\pi^{+}$. The semileptonic nature of the
$\Lambda^{0}_{b}$ decay requires an interface with dedicated generators, in
this case EvtGen. Deeply studied by LHCb, this decay channel includes in its
final state the four charged particle species parameterized in the current
version of Lamarr, namely muons, pions, kaons and protons.

The validation of the Lamarr Tracking modules is depicted in Figure 4 (left)
where the agreement between the $\Lambda^{+}_{c}$ invariant mass distribution
resulting from the ultra-fast paradigm and the one obtained from detailed
simulation proves that the decay dynamics is well reproduced and the
resolution effects correctly parameterized. To show the good performance of
the Lamarr PID models, a comparison between the selection efficiencies for a
tight requirement on a multivariate proton classifier is shown in Figure 4
(right).

Figure 4:  Validation plots for
$\Lambda_{b}^{0}\to\Lambda_{c}^{+}\mu^{-}\bar{\nu}_{\mu}$ decays with
$\Lambda_{c}^{+}\to pK^{-}\pi^{+}$ simulated with Pythia8, EvtGen and Lamarr
(orange markers) and compared with detailed simulation samples relying on
Pythia8, EvtGen and Geant4 (cyan shaded histogram). Reproduced from LHCB-
FIGURE-2022-014.

Comparing the CPU time spent per event by Geant4-based production of
$\Lambda_{b}^{0}\to\Lambda_{c}^{+}\mu^{-}\bar{\nu}_{\mu}$ samples and the one
needed by Lamarr, we estimate a CPU reduction of two-order-of-magnitude only
for the simulation phase. Interestingly, since the generation of $b$-baryons
is exceptionally expensive, Pythia8 becomes the major consumer of CPU
resources in the ultra-fast paradigm. A further speed-up can be reached
reducing the cost for generation, for example using a _Particle Gun_ that
simulates directly the signal particles without going through the high-energy
collisions, not needed since Lamarr parameterizes the detector occupancy. Even
in these physics-simplified settings, the ultra-fast philosophy succeeds in
reproducing thee distributions obtained from detailed simulation
Anderlini:2022ofl .

##  5 Integration with the LHCb simulation framework

To be integrated within the LHCb software stack, the parameterizations
provided by Lamarr need to be queried from a C++ application, running in the
Gaudi framework. Traditional deployment strategies were found to lead to
unacceptably large overheads due to the presence of different multi-threading
schedulers and context switching issues. Hence, a custom deployment strategy
was preferred: models trained with scikit-learn and Keras are converted into
compatible C code using the scikinC toolkit Anderlini:2022ltm , and then
distributed through the LHCb Computing Grid via the CERN VM file-system
(cvmfs) Buncic:2010zz .

The modular layout of Lamarr enables a variety of studies and developments on
the single parameterizations, providing a unique and shared infrastructure for
validation and performance measurements. While crucial for applications within
LHCb, the integration with Gaudi and Gauss makes the adoptions of Lamarr
unappealing for researchers outside of the LHCb community. The SQLamarr
package333Visit https://lamarrsim.github.io/SQLamarr for additional details.
aims to mitigate this problem, providing a stand-alone ultra-fast simulation
framework with minimal dependencies. Based on SQLite3, SQLamarr provides a set
of classes and functions for loading data from physics generators and defining
pipelines from compiled models. An integration between SQLamarr and Gaussino
is currently under investigation with the aim of providing ultra-fast
parameterizations following the experiment-independent philosophy of the
newest LHCb simulation framework, named Gauss-on-Gaussino444Visit
https://lhcb-gauss.docs.cern.ch/Futurev5 for additional details.
Mazurek:2021abc ; Mazurek:2022tlu .

##  6 Conclusion

An evolution of the LHCb software stack and the simulation techniques are
mandatory to meet the upcoming and future demand for simulated samples
expected for $\mathrm{Run\leavevmode\nobreak\ 3}$ and those that will follow.
Ultra-fast-based solutions will play a key role in reducing the pressure on
pledged CPU resources, without compromising unreasonably the description of
the uncertainties introduced in the detection and reconstruction phases. Such
techniques, powered by deep generative models, are provided to LHCb via the
novel Lamarr framework. Well integrated with the physics generators within the
Gauss framework, Lamarr delivers two pipelines according to the charge of the
generated particle. The statistical models for the Tracking and the charged
PID systems have been deployed and validated with satisfactory results on
$\Lambda_{b}^{0}\to\Lambda_{c}^{+}\mu^{-}\bar{\nu}_{\mu}$ decays. Several
models are currently under investigation for the neutral pipeline, where the
translation problem approach offers a viable solution to face the particle-to-
particle correlation problem. Further development of the integration between
Lamarr and the LHCb simulation framework is one of the major ongoing
activities to put the former in production and make its parameterizations
available to the HEP community.

## Acknowledgements

This work is partially supported by ICSC – Centro Nazionale di Ricerca in High
Performance Computing, Big Data and Quantum Computing, funded by European
Union – NextGenerationEU.

## References

  * (1) A.A. Alves, Jr. et al. (LHCb), JINST 3, S08005 (2008)
  * (2) R. Aaij et al. (LHCb), Int. J. Mod. Phys. A 30, 1530022 (2015), 1412.6352
  * (3) M. Clemencic et al. (LHCb), J. Phys. Conf. Ser. 331, 032023 (2011)
  * (4) G. Barrand et al., Comput. Phys. Commun. 140, 45 (2001)
  * (5) T. Sjostrand, S. Mrenna, P.Z. Skands, Comput. Phys. Commun. 178, 852 (2008), 0710.3820
  * (6) D.J. Lange, Nucl. Instrum. Meth. A 462, 152 (2001)
  * (7) J. Allison et al., IEEE Trans. Nucl. Sci. 53, 270 (2006)
  * (8) M. Mazurek, G. Corti, D. MÃller, Comput. Inform. 40, 815â832 (2021), 2112.04789
  * (9) M. Mazurek, M. Clemencic, G. Corti, PoS ICHEP2022, 225 (2023)
  * (10) V. Chekalina et al., EPJ Web Conf. 214, 02034 (2019), 1812.01319
  * (11) A. Maevskiy et al. (LHCb), J. Phys. Conf. Ser. 1525, 012097 (2020), 1905.11825
  * (12) L. Anderlini et al., PoS ICHEP2022, 233 (2023)
  * (13) M. Barbetti (2023), ACAT’22, 2303.11428
  * (14) M. Paganini, L. de Oliveira, B. Nachman, Phys. Rev. D 97, 014021 (2018), 1712.10321
  * (15) C. Krause, D. Shih, Phys. Rev. D 107, 113003 (2023), 2106.05285
  * (16) O. Amram, K. Pedro (2023), 2308.03876
  * (17) F. Ratnikov et al., Nucl. Instrum. Meth. A 1046, 167591 (2023)
  * (18) L. Anderlini et al. (LHCb), J. Phys. Conf. Ser. 2438, 012130 (2023), 2204.09947
  * (19) P. Musella, F. Pandolfi, Comput. Softw. Big Sci. 2, 8 (2018), 1805.00850
  * (20) F. Vaselli et al. (CMS), Tech. rep. (2023), https://cds.cern.ch/record/2858890
  * (21) J. de Favereau et al. (DELPHES 3), JHEP 02, 057 (2014), 1307.6346
  * (22) B.G. Siddi, EPJ Web Conf. 214, 02024 (2019)
  * (23) D. TerjÃk (2020), ICLR’20, 1907.05681
  * (24) A. Rogachev, F. Ratnikov, J. Phys. Conf. Ser. 2438, 012086 (2023), 2207.06329
  * (25) F. Scarselli et al., IEEE Trans. Neural. Netw. 20, 61 (2009)
  * (26) A. Vaswani et al. (2017), NeurIPS’17, 1706.03762
  * (27) S. Brody, U. Alon, E. Yahav (2022), ICLR’22, 2105.14491
  * (28) M. Zaheer et al. (2017), NeurIPS’17, 1703.06114
  * (29) K. Lee et al. (2022), ICLR’22, 2107.04589
  * (30) L. Anderlini, M. Barbetti, PoS CompTools2021, 034 (2022)
  * (31) P. Buncic et al., J. Phys. Conf. Ser. 219, 042003 (2010)