11institutetext: Sardar Patel Institute of Technology, Mumbai, India
11email: {samveg.shah, shubham.thakar, kashish.jain, bhavya.shah,
<EMAIL_ADDRESS>
# A Comparative Study of Machine Learning and Deep Learning Techniques for
Prediction of Co2 Emission in Cars

Samveg Shah 0000-0003-3183-50460000-0003-3183-5046 Shubham Thakar
0000-0003-3101-97350000-0003-3101-9735 Kashish Jain
0000-0002-1115-25070000-0002-1115-2507 Bhavya Shah
0000-0002-4280-97860000-0002-4280-9786 Sudhir Dhage
0000-0001-8100-89410000-0001-8100-8941

###### Abstract

The most recent concern of all people on the Earth is the increase in the
concentration of greenhouse gas in the atmosphere. The concentration of these
gases has risen rapidly over the last century and if the trend continues it
can cause many adverse climatic changes. There have been ways implemented to
curb this by the government by limiting processes that emit a higher amount of
CO2, one such greenhouse gas. However, there is mounting evidence that the CO2
numbers supplied by the government do not accurately reflect the performance
of automobiles on the road. Our proposal of using artificial intelligence
techniques to improve a previously rudimentary process takes a radical tack,
but it fits the bill given the situation. To determine which algorithms and
models produce the greatest outcomes, we compared them all and explored a
novel method of ensembling them. Further, this can be used to foretell the
rise in global temperature and to ground crucial policy decisions like the
adoption of electric vehicles. To estimate emissions from vehicles, we used
machine learning, deep learning, and ensemble learning on a massive dataset.

###### Keywords:

Carbon emission prediction, Ensemble Learning Random Forest XGBoost Neural
Network.

##  1 Introduction

The rise of personal transport in recent years has been accompanied by a sharp
increase in human CO2 emissions across the globe. About a quarter of the EU’s
total CO2 emissions in 2019 were attributable to transportation, the majority
(71.7%) of which originated from vehicle travel. Greenhouse gas emissions have
increased only in the transportation sector over the past three decades,
increasing by 33.5 percent from 1990 to 2019.

Significantly reducing CO2 emissions from transport will not be easy as the
rate of emission reductions has slowed. Although the goal is to reduce
emissions from transportation by 80% by 2050, current projections put that
number much lower, at only 22%. Predictions for the year 2050 indicate a much
lower reduction in transportation-related emissions than what is currently
desired. There is a large disparity between the CO2 emissions of passenger
transport modalities. Sixty-one percent of all carbon dioxide emissions from
automobiles in the European Union are caused by personal vehicles. [1]

According to official laboratory measured monitoring statistics, the average
fuel consumption and CO2 emissions of the European passenger car fleet have
been steadily decreasing. There is mounting evidence that the CO2 numbers
supplied by the government do not accurately reflect the performance of
automobiles on the road. The gap between published figures and real-world
estimations was believed to be 30–40 percent and has been widening over time.
Between 10 and 20 percent of the discrepancy between the stated values and
reality can be attributed to the margins of the current certification method.
For average fleet emissions in 2015, the latter was calculated to be around
40%, or 47.5 gCO2/km. However, it may be as high as 60% or as low as 19%
depending on the day’s traffic. [2]

Even while work is being done to reduce these gaps, there is still a severe
lack of alternate approaches to estimating emissions. This proposal takes a
radical tack, but it’s one that fits the bill given the gravity of the
situation. In order to demonstrate how these cutting-edge methods might offer
a fresh perspective on the problem at hand, we have used machine learning and
deep learning to analyze a sizable collection of vehicle data.

The paper begins with a review of the relevant literature survey in section 2,
then moves on to describe the dataset used and the preprocessing done in
section 3. Then it moves on to describe the methodologies used in section 4.
Then it presents the results in section 5 and ends with the conclusion in
section 6.

##  2 Literature Survey

In paper [3] the authors have created a regression analysis model to estimate
real-world CO2 emissions of light-duty diesel vehicles. Vehicle speed, engine
speed, engine power, and acceleration were some variables that were used for
regression analysis. Further CO2 emission estimators were derived from the
regression equation and were compared with real CO2 emissions.

The authors of [4] have made an effort to use the ensemble technique to
construct a vehicle trip CO2 model by using a significant amount of lab-based
data. They aim to help planners develop better mobility maps that can help
reduce the overall carbon footprint.

The aim of the authors in [5], is to forecast vehicle CO2 emissions per
kilometer and identify an environmentally friendly route that has the lowest
CO2 emissions while still meeting trip time constraints. The theory of vehicle
dynamics is the foundation from which the vehicle CO2 emission model was
constructed.

The authors of paper [6] used data mining techniques to examine the remote
sensing data and estimate the type of fuel and the registration time. Using
decision trees, random forests, AdaBoost, XGBoost, and their fusion models,
the study’s objective was to accurately predict these two important pieces of
information and then apply those predictions to a critical application, namely
the evaluation of automobile emissions.

The variety of different cars and the ongoing advancement of vehicle exhaust
purification technology make it difficult to estimate vehicular transient
emissions accurately, as stressed by paper [7]. They have built CO2 and NOx
transient emission models using a Super-learner model to precisely
characterize the transient emissions of motor vehicles.

[8] considers various tabular datasets and compares the use of deep learning
models and XGBoost on them. They concluded that XGBoost outperforms deep
models on all datasets and also requires much less tuning. However, an
ensemble of XGBoost and deep models performed better than XGBoost alone. This
motivated us to utilize an ensemble of deep models and XGBoost to tackle the
problem of predicting CO2 emissions in cars.

By taking into account both linear and non-linear methods for time-series
forecasting, as well as ensemble and hybrid models by combining components
from various alternatives, the author of the paper [22] surveys the most
recent advancements in supervised machine learning (ML) and high dimensional
models. They also use time series forecasting in the financial and economic
sectors. In [23], the author examines a variety of univariate models,
including AutoRegressive (AR) models, to determine how well they can predict
the future.

The study [24] applied and compared eight different machine learning models to
the well-known M3 time series competition data. The multilayer perceptron and
the regression based on the gaussian process fared the best. The authors of
[25] employed time series data of CO2 69 emissions in India from 1960 to 2017
and the Box- Jenkins ARIMA technique. Based on the forecast, they recommended
five policy recommendations to enhance the environmental circumstances.

In order to anticipate the Air quality index, [26] conducted a prediction-
based study in which they gathered data from Delhi and the National Capital
Region (NCR) of India. They used machine learning models for the same purpose
and evaluated the results using the MAE, MSE, RMSE, and MAPE metrics.

##  3 About The Dataset

The dataset we used was found on the European union website, which had over
90,00,000 rows and 33 different columns. Due to the humongous size of the
dataset, we decided to do some preprocessing first. We eliminated rows with
null values so that it wouldn’t have any adverse effect on our model training.
Then we analyzed the correlation matrix and decided on columns that were
important and not important to the dataset. The dataset we eventually ended up
getting had close to 10,00,000 rows and 18 columns, which have been used to
train the model.

##  4 Methodology

  1. 1.

Preprocessing
The min-max normalization which is a very common technique for normalization
was used on the dataset (excluding the target variable). For each feature, the
minimum value is converted to a 0, the maximum value is converted to a 1, and
all other values are converted to a decimal between 0 and 1. This
normalization technique helps avoid the problem of exploding gradients in the
neural network.

We have used the One hot encoding technique to encode nominal features of the
dataset since there is no numerical relationship between the different
categories. In the case of One hot encoding, each integer value is represented
by a binary vector. The features which have an inherent ranking within the
data points are called ordinal features and ordinal encoding is used for such
categorical features. After splitting the data the dataset has 1,00,000 rows,
while the training dataset has 9,00,000 rows.

Figure 1: Steps for preprocesing

  2. 2.

Models

    1. (a)

Random Forest Regressor

Figure 2: Example of a random forest regressor

      1. i.

Random Forest is an ensemble technique that can perform both regression and
classification tasks by combining multiple decision trees and a technique
known as Bootstrap and Aggregation, or bagging. The basic idea is to use
multiple decision trees to determine the final output rather than relying on
individual decision trees.[12]

      2. ii.

The random forest regressor parameters that produced the best results had
n_estimators (the number of decision trees in the model) set to 250, criterion
(this variable allows you to choose the criterion (loss function) used to
determine model outcomes) set to ”absolute” to represent the mean absolute
error, and max depth (this variable sets the maximum possible depth of each
tree) set to 9.

      3. iii.

The MAE ( mean absolute error ) given by this model was 0.65

    2. (b)

XGBboost

Figure 3: General architecture of XGBboost

      1. i.

XGBoost is a gradient boosting ensemble Machine Learning algorithm based on
decision trees. In prediction issues involving unstructured data, artificial
neural networks outperform all existing algorithms or frameworks (images,
text, etc.). However, for small to medium amounts of structured/tabular data,
decision tree-based algorithms are currently regarded as best-in-class. [11]

      2. ii.

The XGBboost regressor parameters which gave the most optimized results had
the n_estimators (This is the total number of estimators added during model
training.) set to 1000, learning_rate ( The rate at which the XGBoost model
learns during the training phase) set to 0.05, and objective=’reg: linear’ (
It specifies the type of algorithm used to build the model ) set to ’reg:
linear’. These parameters which gave the most optimized results were based on
a randomized search.

      3. iii.

The MAE ( mean absolute error ) given by this model was 2.73

    3. (c)

Neural Network

Figure 4: Sample architecture of neural network

      1. i.

Neural networks, which use neurons to transmit informational bits, closely
resemble the way the human brain functions. Artificial neurons used in neural
networks combine multiple inputs to generate a single output. There are three
layers: Input, Hidden, and Output [10]. The hidden layers perform
transformations on the input and are specialized to produce a definite output.

      2. ii.

We have used the rectified linear activation function with all the layers.
Some advantages of using ReLu include ease of computation, and linear behavior
[27]. The sequential model employed in our neural network architecture has the
following layers:

        1. A.

Dense layer having 128 neurons and ReLu activation function

        2. B.

Dense layer having 512 neurons and ReLu activation function

        3. C.

Dense layer having 512 neurons and ReLu activation function

        4. D.

Dense layer having 512 neurons and ReLu activation function

        5. E.

Dense layer having 256 neurons and ReLu activation function

        6. F.

Dense layer having 256 neurons and ReLu activation function

        7. G.

Dense layer having 1 neuron and ReLu activation function

The last layer represents the output which is the amount of co2 emission

      3. iii.

The Adam optimizer was used to create the model. The results of the Adam
optimizer are generally better than every other optimization algorithm, have
faster computation time, and require fewer parameters for tuning so we decided
to use that. Adaptive Moment Estimation is a technique for optimizing gradient
descent algorithms. The method is extremely efficient when dealing with large
problems involving a large number of data or parameters. It uses less memory
and is more efficient. It appears to be a hybrid of the ’gradient descent with
momentum’ and the ’RMSP’ algorithms. The mean absolute error metric was used.

      4. iv.

The model was trained for 500 epochs with a batch size of 128 and a validation
split of 20

      5. v.

Total trainable parameters were 991873

      6. vi.

The MAE ( mean absolute error ) given by this model was 0.62

    4. (d)

Ensemble Learning

Figure 5: Sample architecture of voting regressor

      1. i.

Ensemble learning is a method of reaching a prediction consensus by combining
the key features of two or more models. Because ensembling reduces the
variance in prediction errors, the final ensemble learning framework is more
robust than the individual models that comprise the ensemble. There are two
primary, linked reasons to choose an ensemble over a single model:
Performance: Compared to a single contributing model, an ensemble can
anticipate events more accurately and perform better overall. Robustness: An
ensemble narrows the prediction and model performance distribution. An
ensemble framework is successful when the contributing models are
statistically diverse. [9]

      2. ii.

The following are some of the benefits of ensemble modeling for neural
networks:

        1. A.

Enhancement of Performance - An ensemble model can yield outcomes with more
generalizability and, in most situations, even greater accuracy by combining
the prediction capacity of its ensemble members.

        2. B.

Enhancement of Reliability - Single, base models frequently suffer from
variance, bias, and noise issues. The ensemble model improves reliability and
robustness by reducing bias and variance and weighting a greater number of
features. Robustness represents low variance, indicating that the model is
unaffected by minor inconsistencies in the training data. Even if the model’s
accuracy does not improve, the ensemble reduces the variance or spread of
prediction.

      3. iii.

A voting regressor, an ensemble meta-estimator that fits multiple base
regressors, each on the entire dataset, is what we utilized. The ultimate
prediction is created by averaging the individual predictions.

      4. iv.

We experimented with 3 ensembles :

        1. A.

Random Forest and Neural Networks: This ensemble had an accuracy of 0.58

        2. B.

Neural Network and XGBoost: This ensemble had an accuracy of 2.2

        3. C.

Random Forest, XGBoost, Neural Networks: This ensemble had an accuracy of 1.74

      5. v.

It can be observed that the ensemble between Random Forest and Neural Network
had the highest accuracy among all the ensembles.

##  5 Results

As described in the previous sections, we have tested our datasets on various
ML algorithms and deep learning techniques. We obtained results in the form of
the Mean Absolute Error which is MAE. The formula for the same is

$\frac{\sum_{i=1}^{n}|y_{i}-x_{i}|)}{n}=MAE$ (1)

Where, yi is the prediction, xi is the true value and n specifies the total
number of data points.

Table 1: Mean Absolute Error Of Different Methods Method | Mean Absolute Error
---|---
Random Forest | 0.65
XGBoost | 2.73
Neural Network | 0.62
Neural Network And XGBoost Ensemble | 2.2
Neural Network And Random Forest Ensemble | 0.58
Neural Network,XGBoost,Random Forest Ensemble | 1.74

##  6 Conclusion

Predicting the emissions of vehicles is a difficult task since it varies
greatly with the size, engine capacity, fuel type, etc. However, it is a
significant problem to solve as the results can help in predicting overall
vehicular emissions and their impact. This can further be used to predict the
rise in global temperature and also anchor important policy decisions such as
shifting to electric vehicles. We applied machine learning, deep learning, and
ensemble learning to a high-volume dataset to solve the problem of predicting
vehicular emissions. We got the best results for our neural network and random
forest ensemble model for which the MAE stands at 0.58. This proposed deep
learning and machine learning ensemble technique can be further tried on other
regression/classification problems.

## References

  * [1] European Parliament. “CO2 Emissions from Cars: Facts and Figures.” Eureporter, June 6, 2022. https://www.eureporter.co/environment/co2-emissions/2022/06/06/co2-emissions-from-cars-facts-and-figures-infographics/.
  * [2] Fontaras, Georgios and Zacharof, Nikiforos and Ciuffo, Biagio. (2017). Fuel consumption and CO2 emissions from passenger cars in Europe – Laboratory versus real-world emissions. Progress in Energy and Combustion Science. 60. 97-131. 10.1016/j.pecs.2016.12.004.
  * [3] Cha, J., Park, J., Lee, H., and Chon, M. S. (2021). A Study of Prediction Based on Regression Analysis for Real-World Co2 Emissions with Light-Duty Diesel Vehicles. International Journal of Automotive Technology, 22(3), 569–577. doi:10.1007/s12239-021-0053-z
  * [4] N. Subramaniam and N. Yusof, ”Modelling of CO2 Emission Prediction for Dynamic Vehicle Travel Behavior Using Ensemble Machine Learning Technique,” 2021 IEEE 19th Student Conference on Research and Development (SCOReD), 2021, pp. 383-387, doi: 10.1109/SCOReD53546.2021.9652757.
  * [5] Weiliang Zeng, Tomio Miwa, Takayuki Morikawa, Prediction of vehicle CO2 emission and its application to eco-routing navigation, Transportation Research Part C: Emerging Technologies, Volume 68, 2016, Pages 194-214, ISSN 0968-090X, https://doi.org/10.1016/j.trc.2016.04.007.
  * [6] Zheng He, Gang Ye, Hui Jiang, Youming Fu, ”Vehicle Emission Detection in Data-Driven Methods”, Mathematical Problems in Engineering, vol. 2020, Article ID 4875310, 13 pages, 2020. https://doi.org/10.1155/2020/4875310
  * [7] Ning Wei, Qijun Zhang, Yanjie Zhang, Jiaxin Jin, Junyu Chang, Zhiwen Yang, Chao Ma, Zhenyu Jia, Chunzhe Ren, Lin Wu, Jianfei Peng, Hongjun Mao, Super-learner model realizes the transient prediction of CO2 and NOx of diesel trucks: Model development, evaluation and interpretation, Environment International, Volume 158, 2022, 106977, ISSN 0160-4120, https://doi.org/10.1016/j.envint.2021.106977.
  * [8] Ravid Shwartz-Ziv, Amitai Armon, Tabular data: Deep learning is not all you need, Information Fusion, Volume 81, 2022, Pages 84-90, ISSN 1566-2535, https://doi.org/10.1016/j.inffus.2021.11.011.
  * [9] E. Bauer and R.. Kohavi. An empirical comparison of voting classification al-gorithms: Bagging, boosting and variants. Machine Learning, 36(1/2):525–536,1999.
  * [10] R. Addaci, R. Staraj, N. Hamdiken, T. Fortaki, ”Neutralization technique model using artificial neural networks”, 2012 24th International Conference on Microelectronics (ICM), pp.1-4, 2012.
  * [11] M. Gumus and M. S. Kiran. Crude oil price forecasting using xgboost.In 2017 International Conference on Computer Science and Engineering(UBMK), pages 1100–1103, Oct 2017
  * [12] Svetnik, A. Liaw, C. Tong, J. Culberson, R. Sheridan, and B. Feuston. Random forest: A classification and regression tool for compound classification and QSAR modeling. Journal of Chemical Information and Computer Sciences, 43:1947–1958, 2003.
  * [13] Luis Pando, Luis E. Garza, ”A Neural Network pressure control approach for automotive Variable Bleed Solenoid”, 18th Mediterranean Conference on Control and Automation, MED’10, pp.1491-1496, 2010.
  * [14] P. W. C. Prasad, A. K. Singh, Azam Beg, Ali Assi, ”Modelling the XOR/XNOR Boolean Functions Complexity Using Neural Network”, 2006 13th IEEE International Conference on Electronics, Circuits and Systems, pp.1348-1351, 2006.
  * [15] Basavaraj Amarapur, ”Neural network based speed control of induction motor”, 2013 Nirma University International Conference on Engineering (NUiCONE), pp.1-6, 2013.
  * [16] Weihong Xu, Zaichen Zhang, Xiaohu You, Chuan Zhang, ”Efficient Deep Convolutional Neural Networks Accelerator without Multiplication and Retraining”, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
  * [17] K. He, X. Zhang, S. Ren and J. Sun, ”Deep residual learning for image recognition”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
  * [18] D. Silver, A. Huang, C. J. Maddison et al., ”Mastering the game of go with deep neural networks and tree search”, Nature, vol. 529, no. 7587, pp. 484-489, Jan 2016.
  * [19] Agarwal, Arun, Kabita Agarwal, and Gourav Misra. ”Is Internet becoming a Major Contributor for Global warming-The Online Carbon Footprint.” Journal of Information Technology and Digital World 2, no. 4 (2020): 217-220.
  * [20] Agarwal, Arun, Kabita Agarwal, and Gourav Misra. ”Is Internet becoming a Major Contributor for Global warming-The Online Carbon Footprint.” Journal of Information Technology and Digital World 2, no. 4 (2020): 217-220.
  * [21] Nirmal, D. ”Design and Efficiency Analysis of Nanocarbon Interconnect Structures.” Journal of Electronics 1, no. 01 (2019): 12-23.
  * [22] Masini, R. P., Medeiros, M. C., and Mendes, E. F. (2021). Machine Learning Advances for Time Series Forecasting. Journal of Economic Surveys. https://doi.org/10.1111/joes.12429
  * [23] Crespo Cuaresma, J., Hlouskova, J., Kossmeier, S., and Obersteiner, M. (2004). Forecasting electricity spot-prices using linear univariate time-series models. Applied Energy, 77(1), 87–106.
  * [24] Ahmed, N. K., Atiya, A. F., El Gayar, N., and El-Shishiny, H. (2010). An empirical comparison of machine learning models for time series forecasting. Econometric Reviews, 29(5), 594–621.
  * [25] Nyoni, T., and Bonga, W. G. (2019). Prediction of co2 emissions in india using arima models. DRJ-Journal of Economics & Finance, 4(2), 1–10.
  * [26] Kumar, R., Kumar, P., and Kumar, Y. (2020). Time Series Data Prediction using IoT and Machine Learning Technique. Procedia Computer Science, 167, 373–381. https://doi.org/10.1016/j.procs.2020.03.240
  * [27] Xavier Glorot, Antoine Bordes, Yoshua Bengio Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:315-323, 2011.


11institutetext: School of Physics and Astronomy, University of Manchester,
M13 9PL, Manchester, United Kingdom22institutetext: Rutherford Appleton
Laboratory, Harwell Campus, Didcot, OX110QX, United Kingdom 33institutetext:
Southampton University, Southampton, United Kingdom

# GPU-based optical photon simulation for the LHCb RICH 1 Detector

Yunlong Li 11 Adam Davis 11 Sajan Easo 22 Keith Evans 11 Evelina Mihova
Gersabeck 11 Marco Gersabeck 11 Lucas George Girardey 2233 Raja Nandakumar 22
Annabelle Jane Pointer 33

###### Abstract

We present the investigation of the use of Opticks, a GPU-accelerated optical
photon interface with the LHCb detector simulation, to improve computation
time of optical photon propagation. The hybrid workflow, combining the
particle simulation package Geant4 and Opticks, offloads optical photon
propagation to GPUs, thereby accelerating the overall simulation process. The
consistency of the results obtained from Geant4 and Opticks simulations is
verified with a simplified LHCb RICH1 detector geometry, demonstrating the
feasibility of the proposed approach. In addition, the ongoing transition to
the NVIDIA OptiX 7 API and re-structuring of Opticks code is discussed within
the context of HEP simulation workflows, with caveats explored.

###### Keywords:

Simulation, Cherenkov radiation, Optical Photon Propagation, GPU, High Energy
Physics, Ray Tracing, OptiX, Opticks, Geant4, LHC, LHCb

##  1 Introduction

Simulation of proton-proton collisions dominates the computing resources of
the LHCb experiment LHCb_2008 . This dominance is illustrated in Figure 1,
which presents the CPU usage at LHCb from January 2022 to presentMathe_2015 ;
Stagni_2017 . The CPU usage is dominated by detailed simulations based on
Geant4 Agostinelli:2002hh ; Allison:2006ve , with a secondary contribution
from fast simulation options, such as ReDecayLHCb-DP-2018-004  and partial
detector simulation. Through explicit measurements, it is known that the
majority of time in LHCb simulation is dominated by the propagation of
particles through the detector within the Geant4 frameworkb_coutri_sim_talk .

Figure 1:  LHCb experiment CPU usage broken down by task from 2022 to present.
As illustrated in blue and orange, the dominant CPU user is the simulation of
collisions. These numbers are from productions of simulated event samples for
LHCb utilising the DIRAC projectMathe_2015  and monitored via the DIRAC web
portal Stagni_2017 . The largest chunk, labelled MC Simulation, represents the
generation and propagation of particles throughout the complete detector
geometry as described in Geant4. The slice labelled MC Fast Simulation is
similar to MC Simulation, but utilises the ReDecay frameworkLHCb-DP-2018-004 .
Finally, MC Reconstruction represents the pattern recognition and track
reconstruction, particle combinations and application of hardware and software
level triggers. Jobs labelled as User are individual submissions, with the
final slice representing all other contributions. Similar dominance of MC
simulation is seen during past data-taking periods.

The projected use of computing resources will only grow as we pass into the
High Luminosity-LHC era, greatly surpassing the resources pledged to the
experiment. As a result, research and development into simulation on diverse
computing architectures is necessary. Graphical Processing Units (GPUs), are
under active investigation, particularly as they have been designed for high-
efficiency parallel processing, allowing for considerable computing cost
reduction and increase in simulation speed. The LHCb collaboration utilises
Geant4 for its fully detailed simulation Agostinelli:2002hh ; Allison:2006ve .
Numerous research and development endeavours have been initiated within the
larger High Energy Physics (HEP) community, targeting the execution of
specific simulation components on GPUs. Figure 2 illustrates the time used in
Geant4 simulations per sub-detector for the latest release of the LHCb
simulation package Gauss LHCb-PROC-2011-006 , highlighting that the
computational resources for simulation are dominated by the calorimeters and
the two Ring Imaging Cherenkov (RICH) detectors. These two systems are
associated with two distinct physical processes: electromagnetic showers and
the transportation of optical photons. The Geant4 collaboration is currently
investigating methods to incorporate GPUs for modelling both processes. The
Celeritas doecode_94866  and ADePT projects Amadio_2023  represent such
development initiatives, aiming to simulate the electromagnetic physics
utilising a GPU. Furthermore, the Opticks package refId0  provides an
interface between Geant4 and the NVIDIA OptiX ray tracing engine to simulate
photon interactions. This paper reports efforts testing the Opticks package
for use by the LHCb simulation framework, further discussed in Sec. 2.

Figure 2: Time spent in each sub-detector relative to the total time for
simulating 100 events, produced by the LHCb performance and regression (PR)
tests. The left part of the plot shows individual sub-detector contributions
and the right part shows sums by detector type, indicating that simulation of
the RICH detectors amount to about one quarter of the total time.

The acceleration of optical photon propagation is not only a necessity for
future LHCb upgrades, but also for numerous neutrino experiments, due to the
significant presence of cosmic muons, which can generate critical background
noise for neutrino detectors such as Daya Bay An_2016  and JUNO JUNO_2016 .
For example, a muon with a typical energy of 200 GeV traversing the JUNO
scintillator is projected to produce tens of millions of optical photons
refId0 . Similarly to the LHCb use case, the propagation of optical photons in
these experiments accounts for over 99% of CPU time and imposes rigorous
memory limitations on the storage of each event refId0 . Consequently, it is
beneficial to explore transfer of optical photon simulation workloads from
CPUs to GPUs, utilising potential programs such as Opticks that can perform
these actions, and which can therefore accelerate the simulation of any
experiment relying on Geant4 optical photon propagation.

###  1.1 OptiX ray tracing engine

The NVIDIA OptiX Parker10OptiX  ray tracing engine constitutes a versatile
programmable system, specifically devised for NVIDIA GPUs and other highly
parallel architectures. As a domain-specific just-in-time compiler, the OptiX
engine operates on the fundamental premise that the majority of ray tracing
algorithms can be executed through a series of programmable operations,
encompassing ray generation, material shading, object intersection, and scene
traversal. OptiX ray tracing pipelines are assembled from a limited set of
user-provided programmes, coded utilising CUDA10.1145/1365490.1365500  — a
parallel computing platform and programming model established by NVIDIA for
general-purpose computing on GPUs. This facilitates the implementation of an
extensive variety of ray tracing-based algorithms and applications, including
interactive rendering, offline rendering, collision detection systems,
artificial intelligence, and scientific simulations, such as sound
propagation. Spatial index data structures, exemplified by the boundary volume
hierarchy (BVH) 10.1145/1572769.1572771 , primarily serve to expedite
intersections between photon paths and geometry structures. OptiX offers a
flexible interface, appropriate for a broad spectrum of applications, to
regulate its acceleration structures and facilitate intersections with any
geometric form. The OptiX acceleration structure system supports adaptable
instancing, which pertains to the low-overhead replication of scene geometry
via referencing the same data multiple times without duplication. This enables
the acceleration of constructing multiple instances of identical geometry,
such as the pixel Multianode Photomultiplier Tubes (MaPMTs) within the LHCb
RICH1 detector.

In August 2019, NVIDIA unveiled OptiX 7, superseding the previous OptiX 6.5
API and offering a low-level, CUDA-centric API. The OptiX 7 API gives explicit
control over memory management, compilation, and launches to the application,
whilst preserving the ray tracing programming model and shader types. The
Opticks package is currently undergoing a transition from the OptiX 6.5 API to
OptiX 7opticks_chep_2023 . Unless indicated otherwise, the subsequent analyses
are performed utilising the OptiX 6.5 API.

###  1.2 Hybrid Geant4 and Opticks workflow

To facilitate the offloading of optical photon propagation in Geant4 to GPUs,
while maintaining the simulation of other particles on CPUs, all elements of
optical photon generation and propagation are converted into a suitable format
and uploaded to the GPU by Opticks. The primary components within the hybrid
workflow encompass detector description including material and surface
properties, as well as optical photon generation and propagation as determined
by optical physics. These aspects are covered in detail in Ref refId0 , and
are summarised here.

####  1.2.1 Geometry translation

During the initialisation phase, Geant4 geometries are initially translated
into the corresponding Opticks geometry buffers. This translation encompasses
the conversion of materials, surfaces, solids, volumes, and sensors, and
construction of complex geometries is accomplished utilising these primitive
shapes through constructive solid geometry (CSG) modelling. The user-provided
Geant4 geometry is then translated into a parallel tree of Opticks nodes. Each
volume boundary is allocated an index uniquely identifying the combination of
four indices representing outer and inner materials and outer and inner
surfaces. Outer/inner surfaces manage inward/outward-directed photons,
enabling the translation of Geant4 border and skin surface functionality.
Surfaces exhibiting a non-zero efficiency property are employed to identify
sensor volumes. To circumvent repetition of geometry translation in each run,
the Opticks geometry is cached within a directory structure for future use.

Upon invoking the visualisation or actual simulation, the Opticks geometry
buffers are further translated into an OptiX geometry node graph, which is
also utilised to configure the resulting acceleration structures. One such
visualisation is presented in Figure 4, which exhibits a ray traced rendering
of the RICH1 detector geometry.

####  1.2.2 Optical photon generation and propagation

We summarise a few key points on optical photon generation and propagation
here for future reference. Optical physics processes, such as scattering,
absorption, scintillator re-emission, and boundary processes, are implemented
in CUDA functions based on the corresponding Geant4 implementations. Rather
than generating photon secondary tracks iteratively, pertinent “genstep”
parameters, including the number of photons to generate and the line segment
along which they should be generated, are collected. These gensteps, in
conjunction with CUDA ports of the Cherenkov and scintillation generation,
facilitate the direct generation of photons on the GPU within the ray
generation programme provided to OptiX. Ultimately, only a small proportion of
the photons detected as hits on the sensor volumes are copied back to the CPU
and incorporated into Geant4 hit collections.

##  2 Test of Opticks with LHCb RICH1 detector

The LHCb detector employs two RICH detectors for particle identification
Amato:494263 ; Collaboration:1624074 . These detectors operate on the
principle of Cherenkov radiation produced by charged particles traversing the
radiator medium at velocities surpassing the speed of light in the radiator.
The emitted light forms a conical pattern, characterised by an angle
$\theta_{c}$ contingent upon the particle’s velocity, given by

$\cos\theta_{c}=\frac{1}{n\beta},$ (1)

where $\beta=v/c$ is the velocity of the particle normalised to the speed of
light in the vacuum and $n$ is the refractive index of the medium, which can
depend on the wavelength of light traversing the medium. By ascertaining the
Cherenkov angle, and combining it with the particle momentum derived from
tracking measurements, a mass hypothesis can be tested for the particle,
facilitating its identification. For this reason, statistical comparisons
between any optical photon propagation offloading and nominal Geant4
techniques are imperative. LHCb is equipped with a pair of RICH detectors,
utilising two distinct radiators: RICH1 employs a $\mathrm{C_{4}F_{10}}$
radiator, possessing a refractive index of 1.0014, to identify tracks in the
momentum range of 2 to 60$\text{\,Ge\kern-1.00006ptV\\!/}c$, while RICH2 is
filled with $\mathrm{CF_{4}}$ gas, with a refractive index of 1.0005,
specifically for momenta as high as 100$\text{\,Ge\kern-1.00006ptV\\!/}c$.
Both RICH detectors incorporate dual sets of mirrors. The primary mirrors are
spherical and are tilted, reflecting Cherenkov photons towards one of the
secondary mirrors. These secondary mirrors are effectively planar, redirecting
the photons outside the LHCb detector acceptance, where the photodetectors are
situated (refer to Fig. 3). This plane aligns with the focal plane of the
corresponding segment of the optical system, causing Cherenkov photons
produced by the same track to focus into a ring on the photon detection plane.

Figure 3:  Schematic view of RICH1 (top) and RICH2 (bottom) detectors and
their optical systems. In Run 2 of the LHC, the silica aerogel in the RICH1
was removed and is not considered further. Taken from LHCb_2008 ; Amato:494263
.

The photodetectors selected for RICH1 consist of MaPMTs, which are structured
as matrices of $8\times 8$ anodes. RICH1 is outfitted with 1-inch MaPMT
modules, each with a pixel size of $2.88\times 2.88\text{\,mm}^{2}$. The
Cherenkov angle resolution for RICH1 is expected to be approximately 0.83 mrad
for a single photon LHCb:2023hlw .

###  2.1 Simplified RICH1 geometry

The full RICH1 system includes support structures, and is at a level of detail
which is unnecessary for validation of the optical photon simulation presented
here. Instead, we use a simplified RICH geometry to validate the performance
of Opticks and check the consistency of Geant4 simulation and hybrid workflow
simulation in a convenient way. This simplified RICH contains only a singular
set of spherical and planar mirrors in the upper section of RICH1, and it does
not incorporate features such as the beam pipe, magnetic shielding, and exit
window — the optical geometry and the radiator are identical to those utilised
in the full RICH1 detector. Detector properties, such as mirror reflectivity
and MaPMT quantum efficiency corrections, are identical to those employed in
the full LHCb RICH1.

Furthermore, to obtain the Geant4 and Opticks simulation results
simultaneously, a Rich_Simplified program based on the Opticks G4Opticks class
has been developed. This program allows direct comparison and validation, as
well as visualisation of the simulation results by propagation of the same
event within the two frameworks.

###  2.2 Measurement of Timing and Statistical Comparisons

The measurement of timing is performed two ways, first using built-in timing
capabilities of the Rich_Simplified program, separately for Geant4 and Opticks
timing, and second, utilising the NVIDIA Nsight tool nvidia_nsight , a package
of debugging and profiling tools designed for NVIDIA GPU-accelerated
applications.

##  3 Results

###  3.1 Rendering of Geometries

The incorporation of the RICH1 and Rich_Simplified geometries are shown in
Figure 4. These renders show the capability of incorporation of arbitrary
geometries within the Opticks framework. However, several iterations were
necessary before an adequate geometry representation was achieved. These
include overlapping volumes as seen by Opticks, incompatible boolean
subtractions between Geant4 based geometries, and incompatibilities between
detection planes of Geant4 and Opticks were discovered and subsequently
updated in the geometry itself. These issues were reported to core developers.

Figure 4: OpenGL rendering of the full RICH1 (top) and simplified RICH1
(bottom) geometry. Both are converted from Geant4 into an Opticks geometry.
The geometry is cut in the centre to allow the visualisation of components.
The colour gradient is artificial and implemented to display the surfaces.

###  3.2 Timing and Profiling

For timing studies we compared the performance using an Intel(R) Xeon(R)
Silver 4210R 2.4 GHz CPU and two NVIDIA Tesla T4 GPUs. The timing results of
the Rich_Simplified programme with Opticks and Geant4 are reported in Figure
5. Reported CPU propagation times are measured using a single core, and GPU
propagation times are presented per GPU used, with two GPUs used in total.

Figure 5: The top figure illustrates the kernel initialisation time
(represented by the red line), photon propagation time (depicted by the blue
line), and total CUDA kernel execution time (shown by the yellow line) as the
initial photon numbers increase. The bottom figure displays the kernel
execution efficiency, defined as the CUDA kernel execution time divided by the
total compute time, as a function of the initial photon numbers.

Figure 5 top shows that there is an underlying overhead of device
initialisation within the Opticks code and is verified via the NVIDIA Nsight
tool. This time corresponds to 35% of the total time of the program for a
single muon propagated through the Simplified RICH geometry. This effect is
further quantified by Figure 5 bottom, which shows the photon propagation time
divided by the total kernel execution time, indicating a measure of efficiency
of the offload process. We find that for the Rich_Simplified program, one
needs to propagate $10^{5}$ photons per kernel execution to ensure at least
95% of the execution time is used to propagate photons.

###  3.3 Statistical Comparisons

The agreement between Geant4 and Opticks simulation results is checked by
running the Rich_Simplified program and performing a statistical comparison of
the results. Test hardware is a single NVIDIA Tesla T4 GPU with Intel(R)
Xeon(R) Silver 4210R 2.40 GHz, 62G CPUs. Figure 6 displays the hits collected
on the MaPMT photocathodes, obtained from a muon passing through the
simplified RICH1 with an energy of 10 GeV. It is worth noting that these hits
are acquired with accounting for quantum efficiency corrections. Upon the
inclusion of quantum efficiency corrections, both Geant4 and Opticks
simulation results yield approximately 60 hits, consistent in position of the
detection plane and also yielding a consistent Cherenkov ring. Figure 7
displays the distributions of hit numbers gathered from 500 events by both
Geant4 and Opticks simulations, fitted using Gaussian functions. The mean
values of collected hits are consistent within one standard deviation, and
both simulations demonstrate a similar Gaussian distribution, suggesting
accurate translations of both geometry and physical processes. The variance in
the mean values stems from two individual points: first, Geant4 allows the
possibility for user defined physics processes to be implemented, which is
taken advantage of in the Rich_Simplified program, for optimisations such as
the elimination multiple bounces of trapped optical photons. The Opticks
package, however, has its own set of physics processes which have been
incorporated within the library. These differing physics processes account for
the shift in the mean position of the peaks. This difference is validated by
removing certain user-implemented physics processes, resulting in a notable
alteration in the observed discrepancies.

Figure 6: The positions of the hits on the MaPMTs’ photocathode are obtained
from the simulation results of Geant4 (in blue) and Opticks (in red). The top
figure displays the positions of the hits in the $x-y$ plane, while the bottom
figure shows the positions in the $y-z$ plane.

Figure 7: The top figure displays the distributions of hit numbers from 500
muons at 10 GeV, simulated by Opticks (in red) and Geant4 (in blue). The means
and standard deviations obtained from Gaussian function fits are indicated on
the plots. The bottom figure presents the distributions of hit numbers from
500 electrons at 10 GeV.

Lastly, the efficiency improvement of Opticks simulation over pure Geant4
simulation is evaluated in two ways: by running a single event with an
increased number of generated photons, achieved by multiplying the expected
number of generated photons by a magnification factor and then using this
number in the simulation, or by running multiple events in a single run. In
this particular instance, approximately 600 photons are generated during a
single event. The results of both methods are presented in Fig. 8, indicating
that Opticks simulation is approximately 10 times faster in the first case and
5 times faster in the second case than pure Geant4 simulation. It should be
noted that the computation times reported only pertain to the duration taken
for optical photon generation and propagation and do not take into account the
time required for geometry translation.

Figure 8: The computational times required by Geant4 (in blue) and Opticks (in
red) simulations are displayed in the figures. The top figure illustrates the
times as the number of photons generated in a single event is increased by a
magnification factor, while the bottom figure depicts the times as the number
of events is increased.

###  3.4 Full RICH1 geometry

Upon verifying the consistency between Geant4 and Opticks simulation results
and confirming Opticks’ acceleration capabilities, a simulation of a single
event through the complete RICH1 geometry is carried out. Figure 9 displays
the hits collected on the PMTs’ photocathode, resulting from random charged
particles passing through the RICH1 detector.

Figure 9: The positions of the hits on the MaPMTs’ photocathode in the $x$-$y$
plane, obtained from the simulation results of Geant4 (in blue) and Opticks
(in red) for the RICH1 geometry. The particle types, directions and initial
energies of the events are generated randomly.

From the collected hits within both Opticks and Geant4, it is clear that
Opticks is also well suited for the use with more than one input particle per
processed event, as multiple Cherenkov rings are visible.

##  4 Discussion

###  4.1 Physics comparison of Opticks with and Geant4

As seen in Figure 9, the overlap of rings of multiple events shows the correct
conversion of the geometry to Opticks. However, the difference between the
distributions, illustrated by the shift in the mean number of hits in Figure 7
and in the areas outside of the rings in Figure 9, illustrates the need for
automatic porting of user defined physics processes to the Opticks framework
in an automatic fashion. Regardless, the work shows that irrespective of
occupancy, the core physics results are consistent. Finally, the linear
scaling of results with number of events in Figure 8 shows optimisation of the
use of GPU resources.

###  4.2 Integration of Opticks with LHCb simulation framework

Integrating Opticks with the LHCb simulation frameworkClemencic_2011 ;
Siddi:2704513 ; Mazurek:2022tlu , presents a considerable challenge, primarily
due to the following two aspects:

First, Opticks relies on numerous external packages, including Geant4,
XercesC, CLHEP, and Boost, as well as other related libraries. These programs
are all required to be specific versions for use in distributed computing
resources. Although Opticks offers an automated command (opticks-full) to
install these external packages, various issues may arise when attempting to
install Opticks on a new machine. To minimise possible conflicts, a bash
script is developed to update, build, and test Opticks, based on the LHCb
Nightly Build System Clemencic:1507655 . This system is a collection of tools
and scripts that automate the build and test tasks for LHCb software, making
the installation of Opticks significantly easier and providing the necessary
foundation for integrating Opticks with the LHCb simulation framework. It can
also provide key insights into compatibility with future software versions.

Second, to deploy Opticks for use in simulation workloads, an interface
between the LHCb core simulation framework and Opticks needs to be developed.
This necessitates major tagged releases of Opticks and the use of consistent
versions of the underlying software across all software projects, highlighted
in the previous paragraph, with the compatibility between all used versions to
be explicitly checked. In the context of the LHCb detector, the optical photon
simulations in the RICH detectors represent one where Opticks would be
invoked. The challenge lies in translating the RICH detectors from the
detector geometry source provided by Gauss into the Opticks geometry buffers,
interfacing other Gauss features such as algorithms, tools, and services with
Opticks, loading external physics lists, and simulating the detector response.
These issues within the integration process require further investigation and
effort. Additionally, the detector construction relies heavily on boolean
volumes, leading to an extremely unbalanced binary search tree. The balancing
of this tree requires either the complete re-design of the geometry or other
tools that can automatically balance the tree to be introduced. Another
possible solution is the export of only the specific geometry elements for
consideration for optical photon workloads by first exporting only these parts
to a parallel geometry within Geant4, then exported to Opticks via GDML or
otherwise.

###  4.3 Migration of Opticks to OptiX 7.0

The code of Opticks is currently in the process of transitioning to the new
NVIDIA OptiX 7 APIopticks_chep_2023 , and as a result, all the GPU code is
currently being restructured. The Opticks GGeo geometry structure will be
replaced by the CSG geometry model, and the interface between Geant4 code and
the Opticks GPU context will be managed by the G4CXOpticks class. Figure 10
presents the renderings of the simplified RICH1 and full RICH1 geometries
using the OptiX 7 API.

Figure 10: Renders of the simplified RICH1 (top) and full RICH1 (bottom)
geometries utilising NVIDIA OptiX 7\. The geometry is modelled using a shared
CPU/GPU geometry model, known as the CSG geometry model, which is designed to
operate with the NVIDIA OptiX 7 API. The geometry is converted using the
CSG_GGeo class and rendered with the assistance of the CSGOptiX class.

At time of study, the full suite of physics processes was unavailable for
OptiX 7, but the rendering capabilities were present, allowing only the report
of rendering results within this paper. While this shows the possibility of
translations of geometries with the same framework but an updated interface,
these geometries will still have unbalanced binary trees, meaning experiments
wishing to use this will need to either optimise their geometries themselves
or incur non-optimal performance. Automatic tools for the detection of such
unbalanced trees may be possible, aiding users in their use of Opticks.

##  5 Conclusion

The integration of Opticks, the GPU-accelerated optical photon simulation
package, with the LHCb detector simulation framework has shown promising
results in terms of efficiency improvement and consistency with Geant4
simulations. The hybrid workflow, which combines Geant4 and Opticks, has
demonstrated the feasibility of offloading the CPU optical photon simulation
to GPUs, improving the overall performance of the simulation process. The
performance characteristics of the Rich_Simplified program provided valuable
insights for accelerating optical photon simulations in the LHCb detector,
particularly in the RICH subsystem, contributing to more efficient data
processing and analysis. We also highlight difficulties encountered and
enumerate future steps necessary for inclusion in full production environments
such as those utilised within LHCb.

## Acknowledgements

We thank S. Blyth for extensive discussions and support with the Opticks
package. We thank M. Kabiri Chimeh the organisers of the UK National GPU
Hackathon 2022 for support. We thank R. Hohl and M. Malenta, our project
mentors during the 2022 UK National GPU Hackathon, for extensive discussions
and support. We also thank the LHCb online team for use of computing resources
related to this work. We thank C. Fitzpatrick and M. Kreps for comments on
early versions of this paper.

This work was supported by UKRI/STFC under grant agreements ST/S000925/1,
ST/W000601/1, and ST/V002546/1; by the Royal Society (United Kingdom) under
grant agreements DH160214 and RGF$\backslash$EA$\backslash$201014.

## References

  * (1) LHCb collaboration, A. A. A. Jr et al., _The LHCb Detector at the LHC_ , Journal of Instrumentation 3 (2008) S08005
  * (2) Z. Mathe, A. C. Ramo, N. Lazovsky, and F. Stagni, _The DIRAC web portal 2.0_ , Journal of Physics: Conference Series 664 (2015) 062039
  * (3) F. Stagni et al., _Dirac in large particle physics experiments_ , Journal of Physics: Conference Series 898 (2017) 092020
  * (4) Geant4 collaboration, S. Agostinelli et al., _Geant4: A simulation toolkit_ , Nucl. Instrum. Meth. A506 (2003) 250
  * (5) Geant4 collaboration, J. Allison et al., _Geant4 developments and applications_ , IEEE Trans. Nucl. Sci. 53 (2006) 270
  * (6) D. Müller, M. Clemencic, G. Corti, and M. Gersabeck, _ReDecay: A novel approach to speed up the simulation at LHCb_ , Eur. Phys. J. C78 (2018) 1009, arXiv:1810.10362
  * (7) B. Coutrier et al., _Ensuring simulation quality in the LHCb Experiment_ , https://indico.jlab.org/event/459/contributions/11525/, 2023
  * (8) M. Clemencic et al., _The LHCb simulation application, Gauss: Design, evolution and experience_, J. Phys. Conf. Ser. 331 (2011) 032023
  * (9) S. R. Johnson et al., _Celeritas_ , [Computer Software] https://doi.org/10.11578/dc.20221011.1, 2022.  doi: 10.11578/dc.20221011.1
  * (10) G. Amadio et al., _Offloading electromagnetic shower transport to GPUs_ , Journal of Physics: Conference Series 2438 (2023) 012055
  * (11) Blyth, Simon, _Integration of JUNO simulation framework with Opticks: GPU accelerated optical propagation via NVIDIA OptiX TM_, EPJ Web Conf. 251 (2021) 03009
  * (12) F. P. An et al., _The detector system of the Daya Bay reactor neutrino experiment_ , Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 811 (2016) 133
  * (13) F. An et al., _Neutrino physics with JUNO_ , Journal of Physics G: Nuclear and Particle Physics 43 (2016) 030401
  * (14) S. G. Parker et al., _OptiX: A general purpose ray tracing engine_ , ACM Trans. Graph. 29 (2010)
  * (15) J. Nickolls, I. Buck, M. Garland, and K. Skadron, _Scalable Parallel Programming with CUDA: Is CUDA the Parallel Programming Model That Application Developers Have Been Waiting For?_ , Queue 6 (2008) 40–53
  * (16) M. Stich, H. Friedrich, and A. Dietrich, _Spatial splits in bounding volume hierarchies_ , in Proceedings of the Conference on High Performance Graphics 2009,  HPG ’09, (New York, NY, USA), 7–13, Association for Computing Machinery, 2009
  * (17) S. Blyth and T. Lin, _Opticks : GPU optical photon simulation using NVIDIA OptiX 7 and NVIDIA CUDA_ , https://indico.jlab.org/event/459/contributions/11811/, 2023
  * (18) LHCb collaboration, S. Amato et al., _LHCb RICH: Technical Design Report_ , https://cds.cern.ch/record/494263, Geneva, 2000
  * (19) LHCb collaboration, _LHCb PID Upgrade Technical Design Report_ , CERN-LHCC-2013-022, LHCB-TDR-014, 2013
  * (20) LHCb collaboration, R. Aaij et al., _The LHCb upgrade I_ , arXiv:2305.10515
  * (21) NVIDIA Corporation, _NVIDIA Nsight systems_ , https://developer.nvidia.com/nsight-systems, 2023
  * (22) M. Clemencic et al., _The LHCb simulation application, Gauss: design, evolution and experience_ , Journal of Physics: Conference Series 331 (2011) 032023
  * (23) B. G. Siddi and D. Muller, _Gaussino \- a Gaudi-Based Core Simulation Framework_ , doi: 10.1109/NSS/MIC42101.2019.9060074
  * (24) M. Mazurek, M. Clemencic, and G. Corti, _Gauss and Gaussino: the LHCb simulation software and its new experiment agnostic core framework_ , PoS ICHEP2022 (2022) 225
  * (25) M. Clemencic and B. Couturier, _A New Nightly Build System for LHCb_ , LHCb-INT-2013-006, CERN-LHCb-INT-2013-006, CERN, Geneva, 2013