# Evaluating How Fine-tuning on Bimodal Data Effects Code Generation

Gabriel Orlanski
<EMAIL_ADDRESS>
Seonhye Yang
<EMAIL_ADDRESS>
Michael Healy
<EMAIL_ADDRESS>
###### Abstract

Despite the increase in popularity of languagemodels for code generation, it
is still unknown how training on bimodal coding forums affects a model’s code
generation performance and reliability. We, therefore, collect a dataset of
over 2.2M StackOverflow questions with answers for finetuning. These finetuned
models have average $pass@k$ improvements of 54.64% and 85.35% on the
HumanEval (Chen et al., 2021) and Mostly Basic Program Problems (Austin et
al., 2021) tasks, respectively. This regime further decreases the number of
generated programs with both syntax and runtime errors. However, we find that
at higher temperatures, there are significant decreases to the model’s ability
to generate runnable programs despite higher $pass@k$ scores, underscoring the
need for better methods of incorporating such data that mitigate these side
effects. The code can be found https://github.com/gabeorlanski/bimodal-code-
generation

Evaluating How Fine-tuning on Bimodal Data Effects Code Generation

Gabriel Orlanski<EMAIL_ADDRESS>Seonhye Yang<EMAIL_ADDRESS>Michael Healy
<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Overview of our approach paper. We first take a pretrained decoder
only language model checkpoint and fine-tune it on mixed modality question and
answers from StackOverflow. We then take this model and evaluate its code
generation performance on HumanEval and Mostly Basic Program Problems (MBPP)
datasets. For each, we generate 200 programs per problem and check their
functional correctness with a set of test cases to calculate their $pass@k$
scores.

The past few years have seen a rapid increase in interest in using pretrained
language models(LM) for program synthesis. Recent works from both Austin et
al. (2021) and Chen et al. (2021) evaluated how large transformer models
(Vaswani et al., 2017) perform on synthesizing Python programs from a natural
language description. Subsequent work has further evaluated how these models
perform in coding competitions (Li et al., 2022) and how pretraining on a
diverse set of programming languages affect downstream performance (Xu et al.,
2022). While much focus has been on how to use GitHub111https://github.com/
data, educational resources such as Stack Overflow
(SO)222https://stackoverflow.com/, where natural language and programming
language occur together in a significantly different setting than the
documentation that occurs in source code, have been under-explored.

Some works have used SO as training data for generation tasks (Yin et al.,
2018; Gao et al., 2020; Orlanski and Gittens, 2021), yet it is not until
recently that there has been interest in utilizing this data for downstream
tasks that require code execution (Fried et al., 2022). However, the effects
of training on a bimodal corpus with both natural language (NL) and
programming language (PL) are still unknown. One potential reason is that
there are no guarantees that the code found has valid syntax, unlike the
source code. Such concerns have inspired works that aim to improve the
reliability of code generation with transformers (Guo et al., 2022; Poesia et
al., 2022). Such solutions, unfortunately, require a significant amount of
engineering. In this paper, we, therefore, aim to answer two questions: (1)
Does fine-tuning a pretrained decoder language model on unstructured bimodal
SO data improve its text-to-code generation performance? (2) How does fine-
tuning on this data impact the model’s ability to generate valid code?

To this end, we first collect and parse over 2.2M python questions with their
answers into a dataset comprising over 1.17B tokens. As our focus is on
generating Python programs, we do not examine how bimodal data with different
programming languages affect downstream performance. We then fine-tune both
CodeParrot 110M (Thoppilan et al., 2022) and GPT Neo (Black et al., 2021)
models on this data. We then evaluate these models on the HumanEval and MBPP
datasets. We find average $pass@k$ improvements of 52.40% and 87.59% for
CodeParrot and GPT Neo, respectively.

Next, we perform ablations in which we remove a modality prior to fine-tuning.
We find that the models fine-tuned on both modalities have, on average, 20.33%
better $pass@k$ scores than those fine-tuned without answer code. Further, the
bimodal models are only 6.11% worse than those trained without natural
language, suggesting that these models can learn without any indication of
modality, but it is not an optimal environment. We conclude with an error
analysis of the generated programs and find that fine-tuning this data
increases the number of errors when evaluating in a zero-shot setting.

##  2 Related Work

Early work on translating NL to PL focused on either generating database
queries (Zelle and Mooney, 1996; Iyer et al., 2019) or domain specific
languages (Dong and Lapata, 2016). Recent years have seen an increased
interest in generating open-ended languages such as Java (Iyer et al., 2018)
and Python (Clement et al., 2020; anne Lachaux et al., 2021). Yin and Neubig
(2017) proposed a neural model in combination with a transition system to
generate abstract syntax trees (Yin and Neubig, 2018). With the advent of the
transformer (Vaswani et al., 2017), the focus has shifted from using code’s
inherent structure to treating it as raw text (Wang et al., 2021; Ahmad et
al., 2021; Clement et al., 2020). Feng et al. (2020) proposed using a masked
language model with bimodal text from the CodeSearchNet challenge (Husain et
al., 2019). Austin et al. (2021) and Chen et al. (2021) both propose using a
decoder language model (Brown et al., 2020) trained on large amounts of source
and web code. Finally, Xu et al. (2022) perform a systematic evaluation of
these models, finding that having some natural language in the training corpus
helps with general code-language modeling.

| HumanEval | MBPP
---|---|---
Model | @1 | @10 | @100 | @1 | @10 | @80
Codex 300Ma | 13.17 | 20.37 | 36.27 | — | — | —
PolyCoder 2.7Bb | 5.60 | 9.80 | 27.70 | — | — | —
InCoder 6.7Bc | 15.20 | 27.80 | 47.00 | 19.40 | — | —
Davinci Codexa ∗ | 36.99 | — | 81.70 | 50.40 | — | 84.40
PaLM Coder 540Bd | 36.00 | — | 88.40 | 47.00 | — | 80.80
CodeParrot 110M | 3.80 | 6.40 | 10.84 | 2.50 | 11.55 | 27.58
+SO | 5.53 | 8.83 | 15.78 | 5.80 | 19.18 | 37.15
+SO -Code | 3.54 | 7.51 | 14.11 | 4.83 | 15.73 | 32.48
+SO -NL | 5.45 | 10.02 | 17.54 | 6.09 | 20.06 | 38.06
GPT Neo 125M | 0.83 | 1.51 | 3.43 | 0.33 | 2.32 | 8.90
+SO | 1.49 | 2.82 | 5.26 | 0.82 | 5.45 | 16.58
+SO -Code | 0.73 | 2.36 | 4.57 | 0.97 | 4.63 | 14.42
+SO -NL | 1.69 | 3.07 | 5.04 | 1.11 | 5.92 | 17.36
Table 1: $pass@k$ Results on both the HumanEval and MBPP task. Best reported
results from three runs with $T\in\\{0.2,0.6,0.8\\}$, and $p=0.95$ and taking
the best values for each $k$. We additionally include results reported by
prior works. The bolded entries are the best value for their respective column
and model in the Model column. aChen et al. (2021) bXu et al. (2022) cFried et
al. (2022) cChen et al. (2021) dChowdhery et al. (2022) ∗ results calculated
by Chowdhery et al. (2022) following the same settings outlined in Chen et al.
(2021). Table 3 has the full results for more $k$ values.

##  3 Method

To utilize the SO data dump333https://archive.org/details/stackexchange
efficiently, we first align the questions with their given answers while
filtering out questions that do not have a Python-related tag. This results in
more than 2.2M questions for nearly 7GB of data. Including the answers from
each question, this translates to approximately 1.17B tokens used for fine-
tuning. For every question, we sort the answers using the following strategy:
if an accepted answer exists, put it first and sort the remaining answers by
their user score. We do not examine other sorting strategies or dataset
filtering based on attributes besides the tags as it is outside this paper’s
scope.

As shown in as shown in Figure 1, we remove all HTML tags from the SO text.
Prior work using SO as a training corpus has explicitly removed code (Ahmad et
al., 2021). We choose not to remove the code as there is evidence that code is
the most important modality contained in SO questions (Orlanski and Gittens,
2021). This line of work is similar to Austin et al. (2021) in that it both
focuses on examining the effects of training LMs on code that is not from
source code. However, they do not remove HTML tags from their training corpus
and use more resources than only SO. Our focus differs in that we limit the
scope to only the case where the training text does not contain modality
markers. We finally pack the title, question body, and answer text into a
single sequence for training.

##  4 Experimental Setup

We fine-tune the CodeParrot (Tunstall et al., 2022) and GPT-Neo (Black et al.,
2021) models with parameter counts 110M and 125M respectively on the SO data
using 4 RTX8000 GPUs with a batch size of 32 and a sequence length of 1024. We
selected these two models as they are both decoder LMs with very different
pretraining data regimes. CodeParrot was only trained on GitHub, whereas GPT-
Neo was trained on the PILE (Gao et al., 2020) which includes a diverse range
of sources. We trained for 25K steps with a starting learning rate of
$5\text{\times}{10}^{-5}$ and a cosine scheduler with 750 warmup steps. We use
the AdamW optimizer (Loshchilov and Hutter, 2019) and the Transformers library
(Wolf et al., 2020).

We then evaluate these models on HumanEval (Chen et al., 2021) and MBPP
(Holtzman et al., 2020) by generating 200 programs for each problem in both
datasets. Performance is then measured by checking if the programs pass given
test cases, the results of which are then used to calculate $pass@k$ (Chen et
al., 2021). Following prior works, we use $k\in\\{1,10,100\\}$ for HumanEval
(Chen et al., 2021) and $k\in\\{1,10,80\\}$ for MBPP (Chowdhery et al., 2022;
Austin et al., 2021).444In Table 3 and for the averages we use
$k\in\\{1,10,50,80,100\\}$.  Unless $k$ is explicitly stated, we report
improvements in $pass@k$ across ablations as the mean percent change of the
$pass@k$ scores for $k\in\\{1,10,50,80,100\\}$. Additionally, for MBPP, we
follow (Austin et al., 2021) by fine-tuning the models on the training split
for 100 steps. For this, we use a batch size of 32, a context window of 1024
tokens, and a learning rate of $1\text{\times}{10}^{-5}$. We prompt the model
with the problem in the task-specific formatting. For HumanEval, this is the
function signature and the docstring, while for MBPP, it is the NL problem
with the three test cases. We follow Austin et al. (2021) in providing the
model an NL prompt in addition to the problem statement. As it is not the
focus of this paper, we do not examine how changing this prompt affects the
performance.

##  5 Results

Table 1 shows our results as well as those reported by recent work on the
HumanEval and MBPP datasets. We find that fine-tuning CodeParrot on SO data
yields average $pass@k$ improvements of 44.95% and 60.85% compared to the
baseline model results on HumanEval and MBPP, respectively. Fine-tuning GPT
Neo on SO data yields a more significant improvement of 65.34% and 109.85% on
HumanEval and MBPP, respectively.

Model | $S$ | $R$ | $T$ | $C$
---|---|---|---|---
CodeParrot |  22.7 |  48.0 |  122.5 |  6.0
+SO |  43.5 |  42.9 |  105.4 |  7.5
-Code |  41.6 |  46.2 |  106.0 |  5.5
-NL |  32.7 |  46.2 |  112.6 |  7.7
GPT Neo |  65.8 |  86.4 |  46.6 |  0.9
+SO |  37.9 |  75.6 |  84.3 |  1.8
-Code |  44.1 |  74.3 |  80.0 |  1.2
-NL |  27.5 |  83.4 |  86.7 |  2.0
Table 2: Average number of programs generated per problem in HumanEval that
have $S$yntax/$R$untime errors, failed the $T$ests, or $C$orrectly solved the
problem. Mean was taken from all programs generated with temperature in
$\\{0.2,0.6,0.8\\}$.  Green values are better than  Red ones.

###  5.1 Removing Modalities

Given that there are multiple modalities present in the SO data, we aim to
quantify the impact that each has on an LM’s program synthesis ability. To
this end, we evaluate the $pass@k$ performance after fine-tuning the pre-
trained models on the SO data with a specific modality removed. We use the
same experimental setup described in section 4 but remove either the Code or
the NL from the answers. Across all runs, we do not modify either the title or
the question body.

The CodeParrot fine-tuned on SO outperforms the same checkpoint fine-tuned on
SO, with the answer code removed by an average of 21.88% and 17.16% on
HumanEval and MBPP, respectively. However, fine-tuning on both modalities
performs worse than the model fine-tuned without answer NL by an average of
7.57% on HumanEval and 3.42% on MBPP. The same is true for GPT Neo, as fine-
tuning on both modalities is 3.74% and 3.42%, worse than when only the answer
code is used on HumanEval and MBPP, respectively. This indicates that while
LMs can learn from unstructured bimodal data, the NL in SO acts as noise.

Figure 2: Breakdown of execution results for programs generated by CodeParrot
across temperatures $\\{0.2,0.6,0.8\\}$. HumanEval has 164 problems and MBPP
has 500 problems. As we generate 200 programs per problem, we have a total of
32,800 and 100,000 programs respectively.

###  5.2 Error Analysis

As shown in Table 2, fine-tuning both CodeParrot and GPT Neo on the full SO
data results in a decrease in the number of programs with runtime errors
generated per problem. A plausible explanation for this is that StackOverflow
is primarily for developers to find help on programming problems and, thus,
specific "triggers" are likely commonly asked about. However, as shown in
Figure 2, the bimodal setting and general lack of structural guarantees cause
an increase in program errors as the temperature increases. When $T=0.8$,
31.13% of the program generated by the model fine-tuned on SO has invalid
syntax. The cause is not the inclusion of the NL, as 16.66% of the programs
generated by the model fine-tuned on SO without NL have syntax errors. One
potential remedy to the situation is to fine-tune the model further on a
solely code dataset such as MBPP, as shown in the right-hand column of Figure
2.

##  6 Conclusion

In this paper, we evaluate the program synthesis abilities of pretrained
decoder language models that have been fine-tuned on bimodal StackOverflow
data. Our results show that this fine-tuning regime does indeed improve
$pass@k$ scores by an average of 70%. We then perform ablations where we
remove each modality, finding that models fine-tuned on the bimodal data
outperforms those fine-tuned on only the NL by an average of 20.33% but is
6.11% worse than those fine-tuned on only the answer code. We conclude with an
error analysis of the generated programs and find that the number of programs
with runtime errors drop by 11.49% in HumanEval but syntax errors rise as
temperature increases. With this in mind, future work should focus on how to
keep the benefits of fine-tuning on StackOverflow while mitigating the
negative side effects.

## Acknowledgment

We thank He He, Angelica Chen, Jacob Austin, and Xaiver Garcia for their
comments and helpful discussions.

## References

  * Ahmad et al. (2021) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Unified Pre-training for Program Understanding and Generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2668, Online. Association for Computational Linguistics.
  * anne Lachaux et al. (2021) Marie anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021\.  DOBF: A deobfuscation pre-training objective for programming languages.  In _Advances in Neural Information Processing Systems_.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021.  Program Synthesis with Large Language Models.  _ArXiv_.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating Large Language Models Trained on Code.  _arXiv:2107.03374 [cs]_.  ArXiv: 2107.03374.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Clement et al. (2020) Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020.  PyMT5: multi-mode translation of natural language and Python code with transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9052–9065, Online. Association for Computational Linguistics.
  * Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016.  Language to logical form with neural attention.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 33–43, Berlin, Germany. Association for Computational Linguistics.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1536–1547, Online. Association for Computational Linguistics.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022.  Incoder: A generative model for code infilling and synthesis.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Guo et al. (2022) Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, and Miltiadis Allamanis. 2022.  Learning to complete code with sketches.  In _International Conference on Learning Representations_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _arXiv preprint arXiv:1909.09436_.
  * Iyer et al. (2019) Srinivasan Iyer, Alvin Cheung, and Luke Zettlemoyer. 2019.  Learning programmatic idioms for scalable semantic parsing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5426–5435, Hong Kong, China. Association for Computational Linguistics.
  * Iyer et al. (2018) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018.  Mapping Language to Code in Programmatic Context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1643–1652, Brussels, Belgium. Association for Computational Linguistics.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022.  Competition-Level Code Generation with AlphaCode.  _arXiv:2203.07814 [cs]_.  ArXiv: 2203.07814.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Orlanski and Gittens (2021) Gabriel Orlanski and Alex Gittens. 2021.  Reading StackOverflow encourages cheating: Adding question text improves extractive code generation.  In _Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)_ , pages 65–76, Online. Association for Computational Linguistics.
  * Poesia et al. (2022) Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022.  Synchromesh: Reliable code generation from pre-trained language models.  In _International Conference on Learning Representations_.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  _ArXiv_ , abs/2201.08239.
  * Tunstall et al. (2022) L. Tunstall, L. von Werra, and T. Wolf. 2022.  _Natural Language Processing with Transformers: Building Language Applications with Hugging Face_.  O’Reilly Media, Incorporated.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  _arXiv preprint arXiv:2109.00859_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2022) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. 2022.  A Systematic Evaluation of Large Language Models of Code.  _arXiv:2202.13169 [cs]_.  ArXiv: 2202.13169.
  * Yin et al. (2018) Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018\.  Learning to mine aligned code and natural language pairs from stack overflow.  In _Proceedings of the 15th International Conference on Mining Software Repositories_ , MSR ’18, page 476–486, New York, NY, USA. Association for Computing Machinery.
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A syntactic neural model for general-purpose code generation.  _arXiv preprint arXiv:1704.01696_.
  * Yin and Neubig (2018) Pengcheng Yin and Graham Neubig. 2018.  TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 7–12, Brussels, Belgium. Association for Computational Linguistics.
  * Zelle and Mooney (1996) John M. Zelle and Raymond J. Mooney. 1996.  Learning to parse database queries using inductive logic programming.  In _AAAI/IAAI, Vol. 2_.

| HumanEval | MBPP
---|---|---
Model | @1 | @10 | @50 | @80 | @100 | @1 | @10 | @50 | @80 | @100
CodeParrot 110M | 3.80 | 6.40 | 9.35 | 10.38 | 10.84 | 2.50 | 11.55 | 23.71 | 27.58 | 29.42
+SO | 5.53 | 8.83 | 13.69 | 14.97 | 15.78 | 5.80 | 19.19 | 32.93 | 37.15 | 39.02
-Code | 3.54 | 7.51 | 12.13 | 13.50 | 14.11 | 4.83 | 15.77 | 28.18 | 32.48 | 34.63
-NL | 5.45 | 10.02 | 14.96 | 16.46 | 17.54 | 6.09 | 20.06 | 34.17 | 38.05 | 39.85
GPT Neo 125M | 0.83 | 1.51 | 2.71 | 3.18 | 3.43 | 0.33 | 2.32 | 6.85 | 8.90 | 9.96
+SO | 1.49 | 2.82 | 4.20 | 4.82 | 5.26 | 0.82 | 5.45 | 13.53 | 16.58 | 18.10
-Code | 0.73 | 2.36 | 3.70 | 4.25 | 4.57 | 0.97 | 4.63 | 11.80 | 14.42 | 15.71
-NL | 1.69 | 3.07 | 4.34 | 4.84 | 5.04 | 1.11 | 5.92 | 14.31 | 17.36 | 18.93
Table 3: Full $pass@k$ Results on both the HumanEval and MBPP task. Best
reported results from three runs with $T\in\\{0.2,0.6,0.8\\}$, and $p=0.95$
and taking the best values for each $k$. We additionally include results
reported by prior works. The bolded entries are the best value for their
respective column and model in the Model column.


# Perspectives on the Calibration of CNN Energy Reconstruction in Highly
Granular Calorimeters

N. Akchurin, C. Cowden, J. Damgov, A. Hussain, and S. Kunori

###### Abstract

We present a study which shows encouraging stability of the response linearity
for a simulated high granularity calorimeter module reconstructed by a CNN
model to miscalibration, bias, and noise effects. Our results also show an
intuitive, quantifiable relationship between these factors and the calibration
parameters. We trained a CNN model to reconstruct energy in the calorimeter
module using simulated single-pion events; we then observed the response of
the model under various miscalibration, bias, and noise conditions that
affected the model input. From these data, we estimated linear response models
to calibrate the CNN. We also quantified the relationship between these
factors and the calibration parameters by regression analysis.

##  1 Introduction

Calibration of depth segmented, non-compensating calorimeters has always
challenged experimenters. Undeterred, some methods have been developed to
calibrate such devices. However, another trend in calorimeter design pushes
the number of readout channels to amounts which warrant sophisticated
reconstruction methods which seem incompatible with previously established
calibration methods. We have started experimenting with applications of
machine learning models - particularly convolutional neural networks or CNNs -
to reconstruct energy from the many signal channels coming from such devices.
The increasing channel count reduces occupancy which reduces the sample size
available in any one channel for calibration; the increasing channel count
also stresses the computational resources available for calibration
activities. For example, a Cu/Si test stand $1\times 1\times
3\;\mathrm{m}^{3}$ with a $50\times 50\times 75$ segmentation scheme contains
$187,500$ cells. If this scheme is extended to a barrel plus endcap
configuration at LHC detector size scales, the combined system contains $\sim
65$ million channels. Whatever the solution may be, methods to calibrate these
devices must possess an efficiency in computation, data usage, and speed of
data collection.

This development raises several serious concerns in relation to calibrating
and extracting reliable physical measurements with confidence. The inherent
non-linearity of CNNs, or any neural network, forces several questions. In
this paper, we confine ourselves to matters related specifically to
calibration, and we leave the tangential questions (e.g. related to data
sources for model training) for another paper. Primarily, we focus on studying
a CNN’s sensitivity to variations of the input data in order to determine the
feasibility of any sort of calibration.

In general, we have two options to correct the CNN’s reconstructed energy. We
can adjust the calorimeter readout channel values, the inputs to the model,
which we shall call pre-model or we can adjust the predicted energy from the
CNN reconstruction, the CNN output, which we shall call post-model.
Historically, experiments have employed linear reconstruction methods, simple
sums of cell responses, resulting in simple multiplicative factors to
calibrate.

A full-size, combined electromagnetic and hadronic high granularity
caloriemter of LHC experimental proportions would have upward of 10’s of
millions of cells. Attempting to calibrate millions of channels via
traditional techniques seems like an unreasonably large task. The degrees of
freedom appear simply too large for experimenters to determine the calibration
parameters with any manageable sized calibration sample. When the calorimeter
is segmented longitudinally, the rear cells will always suffer from low
occupancy and low statistics. Similarly, the physics generating signals at
that depth is different than in the front portion of the calorimeter (assuming
we are not calibrating with muons). As others have reported before [1], this
scenario makes for some unfortunate outcomes. It makes matters worse for
ourselves since we now have to deal with a non-linear energy reconstruction as
well. The time required to collect an adequate sample of cosmic muons to
calibrate scales with the number of cells; consequently, we can expect orders
of magnitude longer to calibrate on cosmics than at current LHC experiments.

We studied the effect of miscalibration and noise on a CNN trained on a Cu/Si
module having a $2\times 2\times 4\;\mathrm{cm}^{3}$ segmentation. In a
previous paper, we described the reconstruction performance of this design
[2]. The CNN we trained operates as a correction to the sum of energy of the
calorimeter cells; the convolutional layers of the CNN consume the raw
calorimeter cell energies and we inject the sum of cell energies at a later
layer of the network. We found this architecture gives us excellent
performance on hadron initiated showers. The data consist of single charged
pions ranging from 0 to 500 GeV simulated by the GEANT4 [3] simulation
framework. We modified the simulated cell energies by applying Equation 1.1
which multiplies the simulated energy in cell $i$ of event $j$ by a
dimensionless miscalibration factor $C_{i}$, it adds a global bias $b$ and a
stochastic noise $\epsilon$ specific to the cell and event. The bias $b$ and
noise $\epsilon$ carry units of MeV to add to the base GEANT4 energy scale.

$\tilde{E_{ij}}=C_{i}E_{ij}+b+\epsilon_{ij}$ (1.1)

We also applied a zero suppression of $0.6\,\mathrm{MeV}$ to the adjusted
energy values which is roughly the energy deposited by a single MIP. The
miscalibration factors are drawn from distributions centered around unity such
that multiplication with simulated energy results in adjustments of the
detector cell’s response. Similarly, noise is drawn from distributions
centered at 0 on an event by event basis such that adding it to the
miscalibrated energy results in a fluctuation around the nominal value. We
draw the miscalibration and noise values from beta distributions, $B$, and
then transform the variables to achieve the desired mean and scale of
variation. We adopted this approach to mitigate the complicating effect of
tails. Equations 1.2 and 1.3 show the transformations of marginal
miscalibration and noise distributions where $s_{c}$ and $s_{\epsilon}$
represent miscalibration and noise scale factors, respectively.

$C=s_{c}(2B_{2,2}-1)+1$ (1.2) $\epsilon=s_{\epsilon}(2B_{2,2}-1)$ (1.3)

These transformations give us distributions of random values in the range
$[1-s_{c},\,1+s_{c}]$ and $[-s_{\epsilon},\,s_{\epsilon}]$ for miscalibration
and noise, respectively.

The correlation of noise and calibration amongst cells results from physical
aspects of a detector design. Lab bench calibration of sub-modules, common
electronics and read out boards for groups of channels, upstream material,
etc. lead to groups of read out channels showing similar noise and response
characteristics. In order to draw random noise values and miscalibration
factors from a joint distribution that captures the variable dependencies, we
turn to vine copulae [4, 5]. Modeling a joint distribution with a copula
allows us to consider the univariate, marginal distributions separately from
the variable inter-dependencies (e.g. correlation). Vine copulas further help
us to structure the variable inter-dependencies by only considering pairs of
partial correlations. We divided the $50\times 50\times 75$ test stand module
into smaller sub-module sections; we chose a vine structure such that cells
within a sub-module are correlated with one another while sub-modules remain
independent. We randomly generated correlation parameters for each sub-module.
This approach allows us to introduce correlation in a way that mimics common
detector readout electronics architectures.

Table 1 lists the parameter values we scanned in a grid to study the CNN
sensitivity. We applied the parameters to modify the GEANT4 simulated cell
energies prior to scoring the CNN. For convenience, we will refer to the
parameter set with the vector $\boldsymbol{\theta}$ with calibration, noise,
and bias scales as the respective elements.

Table 1: The calibration, noise, and bias parameters scanned in a $5\times 8\times 3$ grid. Calibration Scale | Noise Scale [MeV] | Bias [MeV]
---|---|---
0. | 0. | 0.
0.15 | 0.1 | -0.15
0.2 | 0.1 | 0.15
0.25 | 0.3 |
0.3 | 0.4 |
| 0.5 |
| 1. |
| 1.5 |

##  2 Performance Dependencies

We observe the mean CNN response remains linear with respect to the energy of
the initiating particle. Figure 1 shows the mean response as a function of the
initiating pion energy for the calibration parameters $\boldsymbol{\theta}$
indicated in the legend.

Figure 1: The mean CNN response against initiating pion energy for all of the
scanned calibration parameters. This collection of curves gives some visual
justification for our assumption of linearity.

We fit linear models, $\hat{E}\sim aE_{beam}+b$, to the response data. This
parameterization gives us the CNN’s estimate’s dependence on the initiating
pion energy. One expects $a\sim 1$ and $b\sim 0$. Performing this fit for each
point in the parameter scan reveals the reconstruction’s sensitivity to
variations of the input data with respect to the training data. Figure 2
summarizes the results.

Figure 2: Summary of response fits: the subplots show the distributions of
slope and intercept of the CNN response against the calibration, noise, and
bias scales.

In Figure 2, we can see that noise and bias have a stronger influence than the
calibration scale. We can also see the impact of the zero suppression in the
relationship that noise scale has on the fit parameters.

We then fitted independent linear models to the response intercept, $b$, and
to the slope, $a$. We investigated fitting $a$ as dependent upon $b$, but
found it has a statistically insignificant contribution; therefore, we
rejected the dependence. Equations 2.1 and 2.2 enumerate the final estimated
models.

$a=0.963-0.299\cdot(s_{\epsilon}>0.5)\cdot
s_{\epsilon}-0.053\cdot(s_{\epsilon}>0.5)\cdot s_{b}+0.091\cdot s_{b}$ (2.1)
$b=-680\cdot(s_{\epsilon}>0.5)+1098\cdot(s_{\epsilon}>0.5)\times
s_{\epsilon}+2008\cdot(s_{\epsilon}>0.5)\cdot s_{b}$ (2.2)

We find a relatively weak relationship between the bias and the slope compared
to that of noise scale. Interestingly, we observe an interaction between the
noise scale and bias in both fits. This implies that controlling noise should
take the greater share of concern in controlling accuracy. These results are
promising in that we see not only linear responses throughout the scan but we
also see linear relationships between the scanned parameters and response
parameters. This result justifies a multiplicative calibration factor for
post-model calibration; it also implies that we can interpret differences or
changes of calibration constants in terms of relative differences or changes
of properties such as noise scale.

##  3 Discussion

We find the consistent linearity of the CNN response under various input
conditions encouraging for future use of these types of reconstruction models.
The observed linearity implies that we can calibrate the estimated energy by
multiplicative factor and additive bias offsets. Such an approach can
calibrate a single test beam stand or improve observed response uniformity in
a collider-based detector. Although unexpected, a durable linearity of the CNN
response may be a boon to future experiments. Nevertheless, we should remain
somewhat cautious because we have no guarantee that all CNNs will have the
same linear response. We plan for future work to study how a CNN performs
given real test beam data.

## References

  * [1] O. Ganel, R. Wigmans, On the calibration of longitudinally segmented calorimeter systems, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 409 (1) (1998) 621–628.  doi:https://doi.org/10.1016/S0168-9002(97)01337-5.
URL https://www.sciencedirect.com/science/article/pii/S0168900297013375

  * [2] N. Akchurin, C. Cowden, J. Damgov, A. Hussain, S. Kunori, On the use of neural networks for energy reconstruction in high-granularity calorimeters (2021).  arXiv:2107.10207.
  * [3] J. Allison, K. Amako, J. Apostolakis, P. Arce, M. Asai, T. Aso, E. Bagli, A. Bagulya, S. Banerjee, G. Barrand, B. Beck, A. Bogdanov, D. Brandt, J. Brown, H. Burkhardt, P. Canal, D. Cano-Ott, S. Chauvie, K. Cho, G. Cirrone, G. Cooperman, M. Cortés-Giraldo, G. Cosmo, G. Cuttone, G. Depaola, L. Desorgher, X. Dong, A. Dotti, V. Elvira, G. Folger, Z. Francis, A. Galoyan, L. Garnier, M. Gayer, K. Genser, V. Grichine, S. Guatelli, P. Guèye, P. Gumplinger, A. Howard, I. Hřivnáčová, S. Hwang, S. Incerti, A. Ivanchenko, V. Ivanchenko, F. Jones, S. Jun, P. Kaitaniemi, N. Karakatsanis, M. Karamitros, M. Kelsey, A. Kimura, T. Koi, H. Kurashige, A. Lechner, S. Lee, F. Longo, M. Maire, D. Mancusi, A. Mantero, E. Mendoza, B. Morgan, K. Murakami, T. Nikitina, L. Pandola, P. Paprocki, J. Perl, I. Petrović, M. Pia, W. Pokorski, J. Quesada, M. Raine, M. Reis, A. Ribon, A. Ristić Fira, F. Romano, G. Russo, G. Santin, T. Sasaki, D. Sawkey, J. Shin, I. Strakovsky, A. Taborda, S. Tanaka, B. Tomé, T. Toshito, H. Tran, P. Truscott, L. Urban, V. Uzhinsky, J. Verbeke, M. Verderi, B. Wendt, H. Wenzel, D. Wright, D. Wright, T. Yamashita, J. Yarba, H. Yoshida, Recent developments in geant4, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 835 (2016) 186–225.  doi:https://doi.org/10.1016/j.nima.2016.06.125.
URL https://www.sciencedirect.com/science/article/pii/S0168900216306957

  * [4] K. Aas, C. Czado, A. Frigessi, H. Bakken, Pair-copula constructions of multiple dependence, Insurance Mathematics & Economics 44 (2009) 182–198.  doi:https://doi.org/10.1016/j.insmatheco.2007.02.001.
  * [5] T. Bedford, R. M. Cooke, Vines–a new graphical model for dependent random variables, Ann. Statist. 30 (4) (2002) 1031–1068.  doi:10.1214/aos/1031689016.
URL https://doi.org/10.1214/aos/1031689016


# Demonstration of particle tracking with scintillating fibres read out by a
SPAD array sensor and application as a neutrino active target

Matthew Franks  Institute for Particle Physics and Astrophysics (IPA), ETH
Zürich, Ramistrasse, Zürich, 8093, Switzerland  Till Dieminger  Institute for
Particle Physics and Astrophysics (IPA), ETH Zürich, Ramistrasse, Zürich,
8093, Switzerland  Kodai Kaneyasu  Advanced Quantum Architecture Lab (AQUA),
EPFL, Rue de la Maladière, Neuchâtel, 2000, Switzerland  Davide Sgalaberna
Corresponding author. Email<EMAIL_ADDRESS>Institute for Particle
Physics and Astrophysics (IPA), ETH Zürich, Ramistrasse, Zürich, 8093,
Switzerland  Claudio Bruschini  Advanced Quantum Architecture Lab (AQUA),
EPFL, Rue de la Maladière, Neuchâtel, 2000, Switzerland  Edoardo Charbon
Advanced Quantum Architecture Lab (AQUA), EPFL, Rue de la Maladière,
Neuchâtel, 2000, Switzerland  Umut Kose  Institute for Particle Physics and
Astrophysics (IPA), ETH Zürich, Ramistrasse, Zürich, 8093, Switzerland  Botao
Li  Institute for Particle Physics and Astrophysics (IPA), ETH Zürich,
Ramistrasse, Zürich, 8093, Switzerland  Paul Mos  Advanced Quantum
Architecture Lab (AQUA), EPFL, Rue de la Maladière, Neuchâtel, 2000,
Switzerland  Michael Wayne  Advanced Quantum Architecture Lab (AQUA), EPFL,
Rue de la Maladière, Neuchâtel, 2000, Switzerland  Tim Weber  Institute for
Particle Physics and Astrophysics (IPA), ETH Zürich, Ramistrasse, Zürich,
8093, Switzerland  Jialin Wu  Institute for Particle Physics and Astrophysics
(IPA), ETH Zürich, Ramistrasse, Zürich, 8093, Switzerland

###### Abstract

Scintillating fibre detectors combine sub-mm resolution particle tracking,
precise measurements of the particle stopping power and sub-ns time
resolution. Typically, fibres are read out with silicon photomultipliers
(SiPM). Hence, if fibres with a few hundred mm diameter are used, either they
are grouped together and coupled with a single SiPM, losing spatial
resolution, or a very large number of electronic channels is required. In this
article we propose and provide a first demonstration of a novel configuration
which allows each individual scintillating fibre to be read out regardless of
the size of its diameter, by imaging them with Single-Photon Avalanche Diode
(SPAD) array sensors. Differently from SiPMs, SPAD array sensors provide
single-photon detection with single-pixel spatial resolution. In addition,
O(us) or faster coincidence of detected photons allows to obtain noise-free
images. Such a concept can be particularly advantageous if adopted as a
neutrino active target, where scintillating fibres alternated along orthogonal
directions can provide isotropic, high-resolution tracking in a dense material
and reconstruct the kinematics of low-momentum protons (down to 150 MeV/c),
crucial for an accurate characterisation of the neutrino-nucleus cross
section. In this work the tracking capabilities of a bundle of scintillating
fibres coupled to SwissSPAD2 is demonstrated. The impact of such detector
configuration in GeV-neutrino experiments is studied with simulations and
reported. Finally, future plans, including the development of a new SPAD array
sensor optimised for neutrino detection, are discussed.

Keywords: Particle detectors, SPAD array sensor, Scintillating fibres,
Neutrino

##  1 Introduction

Long-baseline neutrino oscillation (LBL) experiments are searching for the
violation of the charge-parity (CP) symmetry in the leptonic sector and aim to
define the neutrino mass ordering thanks to high-intensity neutrino beams and
much larger masses (from several tenths to a few hundred kilotonnes) at the
far detector (FD), located hundreds kilometers away from the neutrino target
production, where the oscillation probability is near maximum [1, 2, 3, 4].
The substantial reduction of the statistical uncertainties will enhance the
sensitivity to the neutrino oscillation parameters. However, in order to
achieve the target sensitivity to the CP-violating phase a drastic reduction
of the systematic uncertainties is necessary. The key is the so-called Near
Detector (ND) that, with a mass that typically ranges from a few hundred
kilograms to several tonnes, is placed only a few hundred meters from the
neutrino production target i.e., before neutrinos can oscillate. Its goal is
the precise detection of neutrino interactions to measure the neutrino flux
and the neutrino-nucleus cross section. The latter is largely affected by the
so-called nuclear effects, such as Fermi motion, binding energy, nucleon-
nucleon correlation also called 2 particle - 2 holes (2p2h) and final-state
interactions, that can introduce a significant smearing and a bias in the
reconstructed neutrino energy and, if not properly modelled, can lead to a
wrong measurement of the neutrino oscillation parameters [5]. New kinematical
analysis methods of the neutrino interaction final state have been developed
in order to distinguish the various nuclear processes and improve the
modelling of the nuclear effects [6, 7]. However, their full exploitation
requires beyond state-of-the-art detectors to identify and reconstruct the
kinematics of all the hadrons in the neutrino interaction final state,
typically protons, neutrons, pions and, possibly, even bound nuclei [8]. This
will be key in reducing the systematic uncertainties at the future neutrino
LBL experiments.

Current and future neutrino detectors, typically based on liquid argon [9, 10,
11, 2] or organic plastic scintillator [12, 13] have improved the state of the
art, by combining particle tracking over the full solid angle with
calorimetry, and reaching proton detection thresholds down to $\sim
300~{}\text{MeV/c}$. Moreover, recently developed 3D granular plastic
scintillator detectors [14], will aim at high-efficiency neutron detection
with energy measurement using the time-of-flight technique [15, 16, 17, 18].
However, none of these detector technologies has a granularity fine enough to
detect low-energy protons that, down to momenta below $200~{}\text{MeV/c}$
leave tracks only a few-mm in length. For such a purpose, High-Pressure Time
Projection Chambers (HpTPC) have been proposed, however, they are still low-
density and require very large magnetised volumes, hence high costs, to reach
a O(1 tonne) mass.

The solution can be found in a scintillating fibre detector (SciFi) used as
neutrino active target that, thanks to a sub-$100~{}\mu\text{m}$ spatial
resolution [19, 20], can easily reconstruct sub-mm proton tracks. SciFi
detectors have been used in neutrino experiments as tracking detectors but not
as neutrino active target [21, 22]. Although similar designs have been
proposed [23], their read out based on CCD shows a relatively high
contamination from noise.

Typically read out with silicon photomultipliers (SiPMs), one limitation is
that to obtain massive detectors necessary to collect enough neutrino
interactions whilst preserving an excellent spatial resolution, a prohibitive
number of electronic readout channels is needed, with consequent large costs.
Hence, a compromise is achieved by coupling a single SiPM (e.g. $1.25\times
0.25\,\text{mm}^{2}$) with a bundle of thinner fibres ($0.25~{}\text{mm}$
diameter) sacrificing the spatial resolution on one spatial coordinate [19].

Recently, a new type of photosensor technology has seen tremendous
improvements. It consists of Single-Photon Avalanche Diode (SPAD) array
sensors. Whilst SiPMs simply count the total number of scintillation photons
detected within its own active area with O(100 ps) time resolution, SPAD
arrays can provide both the position and the time of each single detected
photon with a spatial resolution driven by the size of the single pixel pitch.
Hence, each SPAD pixel can be considered as an independent readout channel.
This is made possible by placing the circuitry for digitisation of single-SPAD
signal on the chip near the pixels, at the expense of a slightly lower fill
factor. Depending on the application that includes LIDAR [24], Non-Line-of-
Sight Imaging [25], and Raman Spectroscopy [26], different SPAD arrays and
SPAD test structures reported in literature demonstrate a pitch down to just
above $2~{}\mu\text{m}$ [27, 28], time gate based readout, as well as multi-
pixel time-stamping using time-to-digital converters (TDCs) with $\sim
50~{}\text{ps}$ timestamp resolution [24]. In [29], SPAD array sensors have
been proposed to readout SciFi. However, particle tracking has not been
demonstrated with prototypes yet.

In this article, we report the demonstration of particle tracking with
SwissSPAD2 [30] and propose a SciFi detector read out with SPAD array sensors
as a cost-effective solution for high-resolution neutrino active target
detectors. The demonstration of particle tracking is obtained by exposing a
bundle of scintillating fibres to 90Sr electrons and reading out with
SwissSPAD2. In the last part of the article, the simulation of a SPAD array
based SciFi neutrino detector is studied both considering the SwissSPAD2
response parameters, as well as an improved design that the authors are
currently developing. Such configuration would allow a reduction of the number
of readout channels, which would scale with the detector surface rather than
the number of scintillating fibres, and, at the same time, preserve the finest
spatial resolution possible in a massive fully-active scintillator volume so
that even low-momentum protons can be tracked. Finally, plans for the future
R&D are discussed.

##  2 Demonstration of readout of scintillating fibres with SwissSPAD2

A demonstrator consisting of SciFi read out with the SwissSPAD2 sensor [30]
has been built and studied. SwissSPAD2 is a $512\times 512$ pixel SPAD array
sensor with a pixel pitch of $16.38~{}\mu\text{m}$, resulting in total surface
of $8.4\times 8.4~{}\text{mm}^{2}$, of which $10.5\%$ is active. The Photon
Detection Probability (PDP) of these pixels peaks at a wavelength of around
500 nm with values of $55\%$, using a $7~{}\text{V}$ excess bias
$V_{\text{ex}}$. During operation, the exposure time of one frame
$t_{\text{gate}}$ can be tuned from $10~{}\text{ns}$ to $10~{}\mu\text{s}$,
and data from the whole array is read out in about $10~{}\mu\text{s}$.

A first prototype of the proposed SciFi detector concept, shown in Fig. 1(a),
was constructed using sixteen $25~{}\text{cm}$ long scintillating fibres
bundled in a $4\times 4$ grid (see Fig. 1(b)), using a machined aluminum
clamp, with around $1~{}\text{cm}$ of fibre protruding from the end of the
clamp to couple to SwissSPAD2.

(a)

(b)

(c)

Figure 1:  (a) SciFi setup used for experiments. Scintillating fibres are
independently coupled to SiPM on their left side end, while on the opposite
side they are bunlded together and coupled to a single SwissSPAD2 sensor. (b)
Close-up photograph of the face of $4\times 4$ scintillating fibre bundle. (c)
Coupling of the scintillating fibre bundle with SwissSPAD2. Black glob top was
used to protect the wire bonding around the active area.

The main goal was the detection of individual tracks created by minimum
ionising particles crossing a scintillating fibre detector read out, for the
first time, by a SPAD array sensor. Its performance was compared to a readout
based on SiPMs, typical for this type of detector which were coupled to the
opposite end of the fibres. Kuraray SCSF-78F $1\times 1~{}\text{mm}^{2}$,
square, single-cladding, plastic scintillating fibres emitting in the blue
around 450 nm [31] were used. The surface of these fibres was polished with
diamond lapping sheet down to $3~{}\mu\text{m}$ grit to ensure a smooth finish
and minimise internal reflections to increase the light yield.

On top of the aluminum clamp a radiation source holder was fastened, allowing
the mounting of a blue flashlight (450 nm nominal wavelength), which was used
to align the SPAD array with the SciFi bundle, tune the respective distance,
and identify the region of pixels detecting the visible photons from each
different scintillating fibre. A 3D printed black plastic collimator with a
100 $\mu$m hole allowed a more precise exposure onto a single column of
fibres, with a reduced light intensity to avoid saturation on the SPAD array.

In a similar manner, the scintillating fibres were also exposed to a 90Sr
radioactive source ($185^{+0}_{-40\%}~{}\text{kBq}$), placed directly on top
of the bundle. Given the presence of a $0.1~{}\text{mm}$ stainless steel foil
directly under the radioactive material inside the source, only electrons with
an energy above 250 keV and a maximum energy of about 2 MeV are expected to
reach the fibres [32]. The range in polystyrene is expected to be longer than
the 4 mm thick bundle for endpoint electrons. The whole SciFi setup, including
the photon readout system (described below), was put on rails, which could be
positioned using a micrometer screw for fine adjustments of the fibre end to
SPAD array distance.

The fibre bundle was coupled on one side with a single SwissSPAD2 as shown in
Fig. 1(c). Although not in contact, the distance between the fibre end and the
SwissSPAD2 active area was less than 0.5 mm. The nominal SwissSPAD2 Photon
Detection Efficiency (PDE) with $V_{\text{ex}}$ equal or larger than 5V is
higher than 4 % at 450 nm [30], the peak of SCSF-78 fibre emission spectrum
[31], and up to about 5% at 500 nm.

A second configuration of SwissSPAD2 was also tested. In order to increase the
effective fill factor, each single pixel was coupled to a microlens. Based on
design simulations, the fill factor can increase up to a factor five [33].
However, given the 0.55 nominal numerical aperture of the used scintillating
fibres [31], the effective fill factor is not expected to achieve its maximum
theoretical value. Later in the text we will name SwissSPAD2-$\mu$lens the
SwissSPAD2 with a microlens on each pixel and SwissSPAD2 the one with naked
pixels.

To monitor the temperature close to the SPAD array, a bluetooth thermosensor
was placed close to the chip and monitored during the data acquisition.

On the opposite end of the SciFi bundle, each individual fibre was polished
and coupled with a SiPM, thanks to 3D printed black optical connectors.
Hamamatsu ceramic-type Multi-Pixel PhotoCounters (MPPCs) with 50 $\mu$m pitch
(S13360-1350CS), for a nominal PDE of 40%, [34] were used. These where mounted
in a custom 3D printed grid as seen on the left in Fig. 1(a).

The whole setup was placed in a dark cabinet and covered with a black cloth to
protect from environmental light.

###  2.1 Setup characterisation

Before exposing the SciFi bundle to 90Sr electrons, the setup was fully
characterised: the position of each fibre on the SPAD array, noisy pixels were
identified, and pixel-to-pixel crosstalk was evaluated.

#### Fibre positions on SPAD array sensor

First, measurements with a collimated, blue flashlight, positioned above the
SciFi bundle were taken to configure and characterise the setup. The same
procedure was followed for both SwissSPAD2-$\mu$lens and SwissSPAD2.

The images in Fig. 3(a) show the 2D projection of the SciFi bundle on to
SwissSPAD2. The contour of each fibre cross-sectional profile was obtained by
fitting the signal count distribution with an error function, as shown in Fig.
3(b). During the tracking analysis, the contour positions will be used to
identify which fibre was crossed by an electron and the centre of the fibre
will identify the particle 2D position.

(a)

(b)

Figure 3: (a) Images created by summing 2,560 frames with the collimated blue
light source positioned over each fibre column. (b) Slice of image of fibre
bundle created by summing all four images in (a). Fits are used to extract the
edges of the fibre bundle and the minima inside these edges are used as the
fibre-to-fibre interfaces.

#### Masking noisy pixels

The unique feature of SPAD array sensors is the possibility to handle each
single pixel independently and exploit both its time and position information.
Such a feature allows to disable noisy or problematic pixels either on the
hardware, firmware or simply during the image analysis, depending on the
particular design of the chip. This feature was not implemented into
SwissSPAD2, therefore in the measurements reported later in Sec. 2.2, 2.3, 2.4
and 2.5 it was sufficient to mask the noisy pixels at the stage of the single
frame analysis.

Since the goal is to detect events where only a few signal counts are
expected, it becomes crucial to minimise the contribution of dark counts (DC),
whilst keeping a sufficiently-high PDE. This would ensure a good tracking
efficiency as well as a low probability to randomly misidentify distributed DC
as a track (fake track). It is worth noting that disabling pixels reduces the
number of DCs as well as the effective fill factor. Hence, a compromise has to
be found.

Over 5,000,000 frames were summed and the total number of counts per pixel was
plotted as a function of the pixel number. The pixels were ordered from lowest
to highest DC rate (DCR), as shown in Fig. 4(a). The median DCR was measured
to be just over 14 counts per second (cps). The noisiest top $5\%$ of pixels,
i.e. with a DCR above 117 cps, were not used in the data frame analysis of the
subsequent measurements. The distribution of the masked pixels in the fibre
bundle region is quite uniform as can be observed in Fig. 4(b). A study
measuring the cross-talk probability between individual SPAD pixels performed
showed a negligible impact. One expects that 0.6 unmasked pixels are triggered
through cross-talk by one of the masked pixels per frame.

(a)

(b)

Figure 4:  (a) Total counts seen in each pixel for $>5~{}\text{M}$ frames in
cps. For the measurement, the noisiest top $5\%$ of pixels were masked. (b)
Distribution of the masked pixels in the fibre bundle region.

###  2.2 Impact of pixel microlenses on SwissSPAD2 fill factor

A first set of measurements with 90Sr electrons was performed with SwissSPAD2.
The single-frame configuration with a $1~{}\mu\text{s}$ gate allowed to detect
and track single electrons produced by 90Sr with a relatively small
contribution from dark counts. The 7 V excess bias provided the maximum PDE,
i.e. slightly above 4% at the scintillating fibre emission peak (450 nm).
2,981,888 frames were collected to characterise the DCR. Then, the same number
of frames was collected while exposing the SciFi bundle to 90Sr electrons.

A second round of measurements was performed with SwissSPAD2-$\mu$lens. This
time 5,012,736 frames where captured in each of the two data runs, one for the
DCR characterisation and one to detect 90Sr electrons.

Fig. 5 shows the distribution of the number of counts per $1~{}\mu\text{s}$
frame in the fibre bundle region after disabling the noisy pixels. The
measurement taken in absence of 90Sr source provides the system background
(BG) (hollow markers), dominated by DCs. The data follows closely a Poisson
distribution, shown with a grey histogram. The distribution of the number of
counts is also shown for data collected while exposing the fibre bundle to
90Sr electroncs for both SwissSPAD2-$\mu$lens (solid dots) and SwissSPAD2
(crosses). In both cases, the number of counts reaches a value definitely
higher than for BG data, a clear signature of electrons traversing the fibre
bundle. The slight shift in the peak of the number of counts per frame with
respect to the BG data is simply given by the simultaneous contribution of BG
and signal electron counts. Moreover, slightly different environmental
conditions can also introduce additional small differences in the DCR. For
instance, a difference in ambient temperature between the two measurements was
recorded ($27.5^{\circ}\text{C}$ for the BG data and $29.0^{\circ}\text{C}$
for the 90Sr data).

As anticipated above, although the fill factor could theoretically increase up
to a factor five for photons perpendicular to the SPAD surface, it is worth
noting that scintillation photons exit the fibre end with a certain angular
distribution [31]. Moreover, systematic effects like microlens-SPAD
misalignment can further reduce the effective fill factor. Hence, the impact
of pixel microlenses on the SPAD PDE shall be studied directly on data. Since
SwissSPAD2-$\mu$lens and SwissSPAD2 were exposed to the same type of electrons
(energy, direction, etc.), the comparison between the position of the
respective spectrum tails allows to claim a relative increase in PDE of around
50%, up to an absolute PDE of approximately 6% for SwissSPAD2-$\mu$lens around
the fibre scintillation emission peak.

Figure 5: Distribution of number of counts per $1~{}\mu\text{s}$ frame
corresponding to the fibre bundle region on SwissSPAD2. The hollow markers
show the distribution obtained from data collected without 90Sr source with
SwissSPAD2-$\mu$lens, known as the background (BG) data. The grey histogram is
a Poisson distribution with $\lambda$ equal to mean counts of the BG data. The
solid dots show the distribution of the number of counts per frame taken with
the 90Sr source positioned above the fibre bundle with SwissSPAD2-$\mu$lens
and the crosses show data obtained with SwissSPAD2 without microlenses. The
data collected with SwissSPAD2 without microlenses and the Poisson
distribution were normalised to 5,012,736 frames.

###  2.3 Tracking of 90Sr electrons

The demonstration of particle tracking with electrons from 90Sr was performed
with SwissSPAD2-$\mu$lens, given the higher PDE. The goal is to detect
unambiguous tracks of electrons in a bundle of scintillating fibres read out
with a SPAD sensor. To our knowledge, such a measurement has never been
reported in literature yet.

A simple algorithm was developed to automatically scan through all frames and
search for vertical track-like events. The pixel-to-fibre mapping, extracted
in Sec. 2.1, was used to count the number of photons detected from each fibre
independently. For this study, a track is defined by three parameters.
Firstly, the number of fibres that the electron travels through, secondly the
number of counts per track, and thirdly the number of counts per fibre. Since
the 90Sr source is positioned at the top of the fibre bundle, an electron is
assumed to always pass through the first row of fibres. The algorithm loops
through all frames and counts the number of hits detected in each fibre region
in the first row. If there is at least one count in any of the fibres, the
algorithm checks the fibres directly below in the same column. If each row
also contains at least one count, the frame is considered as a track-like
event. This is repeated for each column. To remove frames that may contain
multiple tracks, electrons that scatter and travel through multiple columns,
or simply frames that contain too much noise, an SNR analysis was performed.
All counts in the track fibre regions are considered signal and everything
else is considered noise. If the $\text{SNR}>1$, the frame is stored as a
track-like event.

The same track selection algorithm was performed on both the BG and 90Sr+BG
datasets to assess the type and number of fake tracks that arise due to DCs.
This was performed using multiple track selection parameters and is summarised
in Tab. 1. It was found that for events with three fibre hits the probability
that in 90Sr data an electron fake-track was generated from DC is less than 2%
with at least four counts. Similarly, if four fibres were hit, the fake-track
probability was less than 3% with at least four counts. The fake electron
track probability becomes negligible if just a few more counts are required.
It is interesting to note that, simply by considering combinatorial
probability, the chance to misidentify a particle track from dark counts
further reduces near proportionally with the smaller diameter of the
scintillating fibre.

Table 1: Summary of track selection performed on BG and 90Sr+BG datasets. The number of tracks in the table are normalised to same number of frames (5,012,736) for both the BG and 90Sr+BG data. No. fibres | Min. counts | Min. counts | No. of tracks | Misidentification
---|---|---|---|---
per track | per fibre | BG |  90Sr+BG | probability
3 | $3$ | $1$ | $3911$ | $28108$ | $13.9$
$4$ | $1$ | $363$ | $20808$ | $1.7$
$5$ | $1$ | $23$ | $19055$ | $0.1$
$6$ | $1$ | $0$ | $17684$ | $0$
$6$ | $2$ | $0$ | $8160$ | $0$
4 | $4$ | $1$ | $231$ | $8372$ | $2.8$
$5$ | $1$ | $31$ | $7848$ | $0.4$
$6$ | $1$ | $1$ | $7607$ | $>0.1$
$6$ | $2$ | $0$ | $2338$ | $0$

Examples of track-like events from the 90Sr source dataset can be seen in Fig.
6. In the same figure, also the sum of the counts observed from all the
selected vertical tracks is shown.

(a)

(b)

(c)

Figure 6:  Two candidate electron tracks from the 90Sr dataset collected with
SwissSPAD2-$\mu$lens are shown, respectively a vertical through-going track in
6(a) and a vertical stopping track in 6(b). The top figures show the counts in
the $4\times 4$ fibre region on SwissSPAD2-$\mu$lens with single-pixel
binning. The red lines show the fibre contours computed as in Sec. 2.1. The
bottom figures show the same events regrouped with single-fibre binning. In
Fig. 6(c) the flux of 90Sr electrons selected from the vertical-track analysis
is shown. Some non-uniformities between fibres is likely due to slightly
different quality in the treatment of the fibre end.

###  2.4 Comparison with SiPM readout

A subset of the data acquired with SwissSPAD2-$\mu$lens, where electrons
travel vertically through two fibres and come to rest in the third, have been
compared to the same type of events captured by SiPMs.

One Hamamatsu S13360-1350CS MPPC with 50 $\mu$m pitch and maximum nominal PDE
= 40% at 450 nm [35], was precisely coupled to each of the sixteen fibre ends
thanks to a 3D printed black optical connector. Each connector contains a
$1~{}\text{mm}$ thick foam acting like a spring, pushing the SiPM against the
fibre end to minimise the relative distance.

The sixteen SiPMs were connected to individual channels of a 32-channel CAEN
DT5702 readout board [36]. The same 90Sr electron source was positioned
directly above the fibre bundle as before. However, rather than obtaining the
list of pixels with a detected photon in binary format as with the SwissSPAD2
sensors, each SiPM generates an analogue signal whose height is proportional
to the number of photons detected per event in the whole SiPM active area.
Data was acquired for $150\,\text{s}.$

The SiPM signal is given in units of Analogue-to-Digital Converter (ADC). The
pedestal, between 100 and 150 ADC, was subtracted and the gain was extracted
as the conversion factor from the measured number of signal ADCs to the
corresponding number of photoelectrons (PE) by measuring the distance between
the single-PE peaks visible below 500 ADCs. An average gain of 44 ADC/PE was
obtained.

In the SiPM setup a trigger was implemented at the electronics level by
requiring signal in at least one of the fibres on the third layer from top.
Such a trigger was useful to reduce the pile-up of 90Sr electrons and avoid
any event loss during the data taking.

Vertical, stopping tracks were selected from the tracks identified in Sec.
2.3. Exactly the same selection has been applied to the events detected with
SiPMs in order to have a direct comparison with SwissSPAD2-$\mu$lens response
to 90Sr electrons. However, given the different threshold of
SwissSPAD2-$\mu$lens and SiPMs, different cuts on the minimum number of counts
per fibre were applied to obtain approximately the same number of selected
events. This approach avoids to bias the selection that, otherwise, would be
affected by the different PDE values. Requiring at least 2 (14) counts in
SwissSPAD2-$\mu$lens (SiPM) per fibre provided 655 (687) selected events for
one second of exposure. The distributions of the total number of counts per
event per second can be seen in Fig. 7.

Given that the tail ends of both distributions correspond to similar events, a
comparison of the number of detected photons provides an approximate estimate
of SwissSPAD2-$\mu$lens PDE relative the SiPM one. In order to limit the
impact of statistical fluctuations, the relative SwissSPAD2-$\mu$lens and SiPM
PDEs were obtained by integrating the spectrum tail until the right-most 10%
of the events were accounted for. The corresponding ratio of the obtained
number of counts per fibre was used to estimate the ratio between the
SwissSPAD2-$\mu$lens and SiPM overall PDE values, that was found to be about 6
times lower. The same result was obtained also by changing the spectrum tail
integration range. This value is approximately in agreement with the
SwissSPAD2 PDE [30] as well as with the increase in absolute PDE given by the
pixel microlenses, quantified to be about $6\%$ at 450 nm in Sec. 2.2.

Figure 7:  Distribution of the number of counts measured in
SwissSPAD2-$\mu$lens (red) and SiPMs (blue) obtained by exposing the fibre
bundle to 90Sr electrons. Each event corresponds to a selected vertical track
where electrons have travelled through two fibres and come to rest in the
third. The two distributions are normalised to the number of events for one
second exposure and are obtained after a cut on the minimum number of counts
per fibre has been applied to provide about the same number of selected events
(655 for SwissSPAD2-$\mu$lens and 687 for SiPMs).

###  2.5 Setup with scintillating fibre ribbons

As mentioned above, an optimal solution for a SciFi detector capable of 3D
particle tracking by detecting photons produced in the scintillating fibres
and isotropically emitted, whilst keeping the single-fibre resolution in every
spatial coordinate, is to alternate one-fibre-thick layers with different
orientation (e.g. +45/-45/+45/-45/… or 0/+60/-60/0/+60/-60…) resulting in an
XY structure.

A small module compatible with the above setup was constructed to perform
initial tests on a demonstrator with an XY structure. The prototype is shown
in Fig. 8(a), illuminated from the top with the blue light source. The
dimensions of the module are identical to those of the clamp described above,
except that it features a $5\times 5~{}\text{mm}^{2}$ channel to accommodate
different scintillating fibres.

For this demonstrator, BCF-20 fibre ribbons produced by Luxium Solutions
(formerly Saint-Gobain Crystals) [37] were used. The ribbons were produced as
ten, $1~{}\text{m}$ long, $500~{}\mu\text{m}$ diameter fibres with a circular
cross-section, glued together side-by-side. Also, unlike the fibres discussed
in the previous sections, these fibres emit green light with an emission
wavelength of $492~{}\text{nm}$, which in principle better suits the peak PDP
of the SwissSPAD2 sensor.

Similarly to the previous setup, only one plane is read out. For the fibre
layers in the $x$ plane, a single ribbon was cut into five $8~{}\text{cm}$
long pieces. For fibre layers in the $y$ plane, smaller $5~{}\text{mm}$ long
pieces of ribbon were cut and glued using EJ-500 optical cement side-by-side,
perpendicular to the longer lengths. This structure simply fakes the spacing
between layers in an XY configuration. Moreover, it helped to test on a small
scale the gluing of multiple layers before moving to a bigger prototype. This
solid volume of fibre material below the position of the 90Sr source can be
seen in Fig. 8(b) from the point of view of the source.

As before, the fibre ends were polished. The SwissSPAD2-$\mu$lens was placed
at a distance of less than $1~{}\text{mm}$ from the fibre ends as displayed in
Fig. 8(c). The same analysis method described above was performed again for
this module. Fig. 9(a) shows a candidate electron track in a single
$1~{}\mu\text{s}$ exposure frame with single-pixel binning, after masking
noisy pixels and overlaying red circles showing computed fibre contours. The
same frame is shown again in Fig. 9(b) with counts regrouped into single-fibre
bins.

Future R&D plans include a second prototype with XY readout.

(a)

(b)

(c)

Figure 8:  (a) Close-up of XY fibre bundle module illuminated with blue light
source. (b) Top-down view of XY fibre bundle to illustrate perpendicular fibre
layers. (c) XY fibre bundle module pointed at SwissSPAD2-$\mu$lens.

(a)

(b)

Figure 9:  (a) Candidate electron track with single-pixel binning. (b)
Candidate electron track with single-fibre binning, where the three counts
falling outside the fibre contours (see Fig. (a)) are not included in the
image.

##  3 Scintillating fibres and SPAD arrays for precise neutrino detection

In the previous sections, the tests with SwissSPAD2 provided an initial
demonstration of a novel particle detector configuration with scintillating
fibres read out by SPAD array sensors. These successful tests show the path
towards a drastic improvement in the technology of ultra-fine granularity
scintillating detectors, thanks to the $\sim 10~{}\mu\text{m}$ spatial
resolution of monolithic SPAD array sensors, combined with minimal deployment
of electronics readout channels, which scales with the covered surface and no
longer by the number of scintillating fibres. It was also observed that the
fine granularity also minimises the probability to misidentify a particle
track from dark counts, and that it would further reduce with the diameter of
the scintillating fibres. In addition, the sensors can be cooled, lowering the
DCR by up to an order of magnitude for 20 degrees Kelvin [38, 39, 40]. To
achieve miniature pixels, SwissSPAD2 was designed without timestamping on-
chip, however, SPAD arrays with $\sim$ 50 ps single-photon timestamp
resolution exist [40]. This would allow the use of temporal correlation of
hits to further reduce the amount of DCs inside the data sample without the
need of using very short time gates.

A back-of-the-envelope calculation, based on the DCR of SwissSPAD2 [30],
results in an expected DCR of less than 2 DCs per 1000 250 $\mu$m fibres in a
10 ns time window at room temperature. Therefore, the background due to
electronic noise can be considered negligible, which matches the measurements
reported above.

In this section the application of this novel detector concept as well as its
optimal configuration for the detection of neutrinos is studied using
simulations. Such a setup would allow the detection of neutrino interactions
with unprecedented spatial and time resolution. The main advantages are
expected for the following measurements, that can highly profit from a superb
spatial resolution combined with an excellent particle identification:
detection of low-momentum particles (below 300 MeV/c), such as protons or
nuclear clusters, crucial for the understanding of nuclear effects; improved
charge and momentum reconstruction of final state particles; high purity
$\nu_{e}$ and $\bar{\nu}_{e}$ detection with minimal contamination from
$\gamma\rightarrow e^{+}e^{-}$ events, typically the dominant background; and
detection of final state neutrons, in particular low-energy ones that can
deposit energy near the neutrino interaction vertex. All these measurements
will be crucial for the systematic reduction in the future LBL experiments.
Another application of such a novel detector configuration would be the
detection of $\nu_{\tau}$ with precise identification of the $\tau$ decay
vertex around 1 mm from its production point with timing information [41],
which is, however, not studied in this work.

A setup assuming either an 6% or 20% overall PDE SPAD array sensor
instrumenting a large active neutrino target was simulated. To achieve such a
higher PDE one may: adopt fibres better matching to the SPAD peak PDP, as in
Sec. 2.5; read out from both ends of the scintillating fibers; and improve the
SPAD architecture by increasing the fill factor. The target consists of 2,000
XY-layers of 2 m long and 250 $\mu\text{m}$ thick fibres arranged in layers of
8,000 fibres each, resulting in a $2\text{m}\times 2\text{m}\times 1\text{m}$
block. Using NEUT [42], 100,000 charged current muon neutrino ($\nu_{\mu}$)
interactions following the T2K experiment energy flux [43] were simulated. The
final state particles were propagated through the detector geometry
implemented in Geant4 [44, 45, 46]. The particle energy deposition was
simulated and corrected for Birks’ quenching [47]. The conversion from the
deposited energy in the fibres (MeV) to the number of detected photons was
validated by simulating and comparing with data from the 90Sr tests described
in Sec. 2.4. Unfolding the 40% PDE of the SiPMs and applying the 20% (6%) PDE
of the assumed SPAD array sensors, an effective light yield of about 100 (30)
$\gamma$/MeV was determined.

Finally, to determine the number of photons produced at different positions
along the fibres and detected at the SPAD array, the light yield was corrected
for the attenuation length [31]. As discussed above, random noise is expected
to be extremely small, in particular after applying the track algorithm
selection. Thus, it is neglected in this study. An example of a $\nu_{\mu}$
CCQE event can be seen in Fig. 10. The proton Bragg peak, whose stopping power
can be as high as 5–10 times that of a minimum ionizing particle (MIP), is
well visible even for a relatively low PDE. This could make SPAD array sensors
a viable solution for the detection of low-momentum protons already with
existing technologies. Using the high PDE SPAD array sensor, the MIP can be
tracked more precisely, even showing delta-electrons along the track.

(a)

(b)

Figure 10:  Simulated $\nu_{\mu}$ Charged Current Quasi Elastic (CCQE)
interaction in the centre of the SciFi detector, undergoing final state
interactions, producing extra protons. The colour shows the number of photons
detected from the scintillating fibres on the XZ projection with SPAD array
sensors whose PDE is equal to (a) 6% and (b) 20%. Neutrinos are simulated
along the Z axis. Tracks from both the muon and three stopping protons can be
seen. The latter show a higher energy deposition due to the Bragg peak at the
end of their tracks.

A “pseudo-reconstruction” was implemented to study the physics potential of
such a detector configuration. A “hit” is observed if at least one photon is
detected by the SPAD array from a given fibre. A track is reconstructed if it
contains at least five hits, three in one projection and two in the orthogonal
one. Moreover, in order to account for the overlap between different particle
tracks in the tracking efficiency, the shorter track is assumed to be
reconstructed if its last hit is separated by at least 2 fibres from the
nearest track. The results of this study show that protons down to 150 MeV/c
can be reliably detected, as shown in Fig. 11(b), thanks to the fine
granularity of plastic scintillating fibres. The notable drop in efficiency at
$\cos\theta\sim 0$ is due to protons that, travelling along a single fibre, do
not fulfill the reconstruction criteria. In Fig. 11(c) the proton tracking
efficiency is plotted as a function of the proton momentum and superimposed to
the truth proton spectrum corresponding to $\nu_{\mu}$ CC interactions without
pions in the final state. A SciFi detector gives access to the full spectrum
of protons produced by neutrino-nucleus interactions and becomes a tool to
resolve the nuclear effects and constrain the major systematics at the future
LBL neutrino oscillation experiments. It is interesting to note that, given
the high stopping power, low-momentum protons can be efficiently reconstructed
even with SPAD array sensors with a PDE down to about 6%. Studies
investigating particle identification using the ionisation density along the
path, charge identification and overall calorimetry are in progress. However,
a very good proton purity up to 95% can be achieved thanks to the Bragg peak
which is much more pronounced compared to that of muons and pions.

(a)

(b)

(c)

Figure 11:  (a) Simulated proton tracking efficiency as a function of the
proton momentum and direction (cosine of the angle with respect to the
neutrino direction) with a 6% PDE SPAD array, similar to the one used in the
measurements in the previous sections. (b) Simulated proton tracking
efficiency as a function of the proton momentum and direction (cosine of the
angle with respect to the neutrino direction) with a 20% PDE SPAD array, as
proposed in this paper. (c) Proton tracking efficiency overlaid to the
momentum distribution of protons produced by $\nu_{\mu}$-carbon interactions
without pions in the final state. The proton momentum distribution is broken
down by the different neutrino interaction modes, i.e. CCQE, 2p2h and CC with
one pion. The simulation was performed with NEUT [42] using the T2K neutrino
flux [43].

Another promising study performed is the rejection of $\gamma\to e^{+}e^{-}$
events, a major background to $\overset{\textbf{(---)}}{\nu}_{e}$
interactions. Due to the high granularity of this type of detector, it is
possible to resolve the initial tracks of both the electron and the positron,
even in relatively low energy gamma conversion events. In a 0.2 T magnetic
field, it was found that for $\gamma\rightarrow e^{+}e^{-}$ events between 0.1
and 0.5 GeV, the major background in the detection of $\nu_{e}$-nucleus
interactions, only about 5% cannot be fully reconstructed and properly
identified. Such a fraction is largely dominated by Compton scattering where
the observable is just a single electron track.

##  4 Conclusions

We proposed a neutrino active target detector consisting of scintillating
fibres read out with SPAD array sensors to detect and track particle events
produced by neutrino interactions, such as low-momentum protons or electrons
and positrons from photon conversion, key for the reduction of the systematic
uncertainty at the future long-baseline neutrino oscillation experiments.

SPAD array sensors are monolithic imaging devices where each SPAD pixel acts
as an effectively independent read out channel, allowing to detect single
photons with a spatial resolution below $10~{}\mu\text{m}$ with minimal impact
from random noise and, depending on the chosen readout strategy, also the time
of arrival with a time resolution below 100 ps. One clear advantage is that
the number of electronic channels scales with the detector surface rather than
the number of scintillating fibres, with a potentially drastic reduction of
the costs, despite an extremely fine detector granularity.

In this work we provided the first proof of concept of particle tracking with
a state-of-the-art SPAD array sensor (SwissSPAD2) and analysed the main
parameters to be optimised for future applications in neutrino experiments.

The result of this work is being used to develop a new SPAD array sensor, on
whose design the authors are currently working. The current PDE is relatively
low compared to SiPMs, due to the lower fill factor, and is the main parameter
that needs to be improved, together with the single-pixel time resolution. On
the other hand, it is worth noting that even a PDE below 10% would allow to
efficiently track protons below 300 MeV/c, thanks to the high-stopping power.
The authors are now also moving towards a more advanced prototyping of an XY
SciFi configuration, still read out with SPAD array sensors.

Simulation studies demonstrated the potential of SciFi in the detection of
particles produced by neutrino interactions, in particular those leaving very
short tracks that are invisible to state-of-the-art tonne-scale neutrino
detectors.

Finally, it is interesting to note that one may alternate SciFi modules with
thicker volumes of coarser 3D-segmented scintillator, like the one in [14], to
obtain a massive detector where scintillating fibres can be used as neutrino
active targets as well as for improving the reconstruction of particles
produced by interactions in the coarser granularity part. Such a configuration
could be optimal for combining high-resolution tracking with the efficient
detection of neutrons that, as shown in [15, 17], require large volumes of
dense hydrocarbon-like active material, or of electron neutrinos ($\nu_{e}$),
that typically contaminate only a few percent of the $\nu_{\mu}$ dominated
neutrino beam. Moreover, like already done with nuclear emulsions [48, 49],
single layers of XY SciFi could be alternated with very thin volume of
inactive target material to precisely measure neutrino interactions also in
nuclei other than carbon or hydrogen, like water, but with the advantage of an
excellent time resolution.

## Statements and Declarations

The authors declare that there are no conflicts of interest related to this
article. For the sake of transparency, the authors would like to disclose that
(i) Edoardo Charbon holds the position of Chief Scientific Officer of
Fastree3D, a company making LiDARs for the automotive market, and that (ii)
Claudio Bruschini and Edoardo Charbon are co-founders of Pi Imaging
Technology. Both companies have not been involved with the paper drafting, and
at the time of writing have no commercial interests related to this article.

## Acknowledgments

This work was supported by the SNF grant PCEFP2_203261, Switzerland. This
research was also partially supported by the Swiss National Science Foundation
(grant 20QT21_187716 Qu3D “Quantum 3D Imaging at high speed and high
resolution”).

## References

  * [1] K. Abe et al.  Letter of Intent: The Hyper-Kamiokande Experiment — Detector Design and Physics Potential —.  2011\.
  * [2] B. Abi et al.  Long-baseline neutrino oscillation physics potential of the DUNE experiment.  Eur. Phys. J. C, 80(10):978, 2020.  10.1140/epjc/s10052-020-08456-z.
  * [3] K. Abe et al.  Constraint on the matter–antimatter symmetry-violating phase in neutrino oscillations.  Nature, 580(7803):339–344, 2020.  10.1038/s41586-020-2177-0.
  * [4] M. A. Acero et al.  Measurement of the $\nu_{e}-$Nucleus Charged-Current Double-Differential Cross Section at $\left<E_{\nu}\right>=$ 2.4 GeV using NOvA.  6 2022.
  * [5] A. M. Ankowski.  Effect of the charged-lepton’s mass on the quasielastic neutrino cross sections.  Phys. Rev. C, 96(3):035501, 2017.  10.1103/PhysRevC.96.035501.
  * [6] X. Lu et al.  Measurement of nuclear effects in neutrino interactions with minimal dependence on neutrino energy.  Phys. Rev. C, 94(1):015503, 2016.  10.1103/PhysRevC.94.015503.
  * [7] X. Lu et al.  Reconstruction of Energy Spectra of Neutrino Beams Independent of Nuclear Effects.  Phys. Rev. D, 92(5):051302, 2015.  10.1103/PhysRevD.92.051302.
  * [8] A. Ershova et al.  Study of final-state interactions of protons in neutrino-nucleus scattering with INCL and NuWro cascade models.  Phys. Rev. D, 106(3):032009, 2022.  10.1103/PhysRevD.106.032009.
  * [9] R. Acciarri et al.  Design and Construction of the MicroBooNE Detector.  JINST, 12(02):P02017, 2017.  10.1088/1748-0221/12/02/P02017.
  * [10] P. Abratenko et al.  ICARUS at the Fermilab Short-Baseline Neutrino program: initial operation.  Eur. Phys. J. C, 83(6):467, 2023.  10.1140/epjc/s10052-023-11610-y.
  * [11] R. Acciarri et al.  Construction of precision wire readout planes for the Short-Baseline Near Detector (SBND).  JINST, 15(06):P06033, 2020.  10.1088/1748-0221/15/06/P06033.
  * [12] P. A. Amaudruz et al.  The T2K Fine-Grained Detectors.  Nucl. Instrum. Meth. A, 696:1–31, 2012.  10.1016/j.nima.2012.08.020.
  * [13] Y. Abreu et al.  Performance of a full scale prototype detector at the BR2 reactor for the SoLid experiment.  JINST, 13(05):P05005, 2018.  10.1088/1748-0221/13/05/P05005.
  * [14] A. Blondel et al.  A fully-active fine-grained detector with three readout views.  Journal of Instrumentation, 13(02):P02006–P02006, 2018\.  10.1088/1748-0221/13/02/p02006.
  * [15] L. Munteanu et al.  New method for an improved antineutrino energy reconstruction with charged-current interactions in next-generation detectors.  Phys. Rev. D, 101(9):092003, 2020.  10.1103/PhysRevD.101.092003.
  * [16] K. Abe at al.  T2K ND280 Upgrade - Technical Design Report.  2019\.
  * [17] S. Gwon et al.  Neutron detection and application with a novel 3D-projection scintillator tracker in the future long-baseline neutrino oscillation experiments.  Phys. Rev. D, 107(3):032012, 2023.  10.1103/PhysRevD.107.032012.
  * [18] A. Agarwal et al.  Total neutron cross-section measurement on CH with a novel 3D-projection scintillator detector.  Phys. Lett. B, 840:137843, 2023.  10.1016/j.physletb.2023.137843.
  * [19] C. Joram et al.  LHCb Scintillating Fibre Tracker Engineering Design Review Report: Fibres, Mats and Modules.  2015\.  LHCb-PUB-2015-008, CERN-LHCb-PUB-2015-008.
  * [20] A. Papa et al.  The Mu3e scintillating fiber detector R&D.  Nucl. Instrum. Meth. A, 1050:168099, 2023.  10.1016/j.nima.2023.168099.
  * [21] E. Eskut et al.  Final results on $\nu_{\mu}\rightarrow\nu_{\tau}$ oscillation from the CHORUS experiment.  Nucl. Phys. B, 793:326–343, 2008.  10.1016/j.nuclphysb.2007.10.023.
  * [22] R. Albanese et al.  Observation of collider muon neutrinos with the SND@LHC experiment.  5 2023.  CERN-EP-2023-092.
  * [23] P. Annis et al.  A new vertex detector made of glass capillaries.  Nucl. Instrum. Meth. A, 386:72–80, 1997.  10.1016/S0168-9002(96)01098-4.
  * [24] C. Zhang et al.  A 30-frames/s, $252\times 144$ SPAD Flash LiDAR With 1728 Dual-Clock 48.8-ps TDCs, and Pixel-Wise Integrated Histogramming.  IEEE J. Solid State Circuits, 54(4):1137–1151, 2019.  10.1109/JSSC.2018.2883720.
  * [25] D. Faccio, A. Velten, & G. Wetzstein.  Non-line-of-sight imaging.  Nature Reviews Physics, 2(6):318–327, 2020.  10.1038/s42254-020-0174-8.
  * [26] C. Bruschini et al.  Single-photon avalanche diode imagers in biophotonics: review and outlook.  Light Sci Appl, 8:87, 2019.  10.1038/s41377-019-0191-5.
  * [27] K. Morimoto & E. Charbon.  High fill-factor miniaturized SPAD arrays with a guard-ring-sharing technique.  Opt. Express, 28(9):13068–13080, 2020.  10.1364/OE.389216.
  * [28] S. Shimada et al.  A SPAD Depth Sensor Robust Against Ambient Light: The Importance of Pixel Scaling and Demonstration of a 2.5$\mu$m Pixel with 21.8% PDE at 940nm.  In 2022 International Electron Devices Meeting (IEDM), pages 37.3.1–37.3.4, 2022.  10.1109/IEDM45625.2022.10019414.
  * [29] P. Fischer, R. K. Zimmermann, & B. Maisano.  CMOS SPAD sensor chip for the readout of scintillating fibers.  Nucl. Instrum. Meth. A, 1040:167033, 2022.  10.1016/j.nima.2022.167033.
  * [30] A. C. Ulku et al.  A $512\times 512$ SPAD Image Sensor With Integrated Gating for Widefield FLIM.  IEEE Journal of Selected Topics in Quantum Electronics, 25:1–12, 2019.  10.1109/jstqe.2018.2867439.
  * [31] Kuraray catalogue - Plastic Scintillating Fibers.  http://kuraraypsf.jp/psf/sf.html.
  * [32] T. Roy, F. Tessier & M. McEwen.  A system for the measurement of electron stopping powers: proof of principle using a pure $\beta$-emitting source.  Radiation Physics and Chemistry, 149:134–141, 2018.  10.1016/j.radphyschem.2018.03.006.
  * [33] C. Bruschini et al.  Challenges and prospects for multi-chip microlens imprints on front-side illuminated SPAD imagers.  Opt. Express, 31(13):21935–21953, 2023.  10.1364/OE.488177.
  * [34] https://www.hamamatsu.com/eu/en/product/optical-sensors/mppc/index.html.
  * [35] Hamamatsu.  MPPC (multi-pixel photon counter) S13360 series, 2022.  https://www.hamamatsu.com/content/dam/hamamatsu-photonics/sites/documents/99_SALES_LIBRARY/ssd/s13360_series_kapd1052e.pdf.
  * [36] CAEN catalogue.  https://www.caen.it/products/dt5702/.
  * [37] Luxium Solutions.  https://luxiumsolutions.com/.
  * [38] E. A. Webster et al.  A Single-Photon Avalanche Diode in 90-nm CMOS Imaging Technology With 44% Photon Detection Efficiency at 690 nm.  IEEE Electron Device Letters, 33(5):694–696, 2012.  10.1109/LED.2012.2187420.
  * [39] X. Hesong et al.  Design and characterization of a p$+$/n-well SPAD array in 150nm CMOS process.  Opt. Express, 25(11):12765–12778, 2017.  10.1364/OE.25.012765.
  * [40] F. Gramuglia et al.  A Low-Noise CMOS SPAD Pixel With 12.1 Ps SPTR and 3 Ns Dead Time.  IEEE Journal of Selected Topics in Quantum Electronics, 28(2: Optical Detectors):1–9, 2022.  10.1109/JSTQE.2021.3088216.
  * [41] K. Kodama et al.  Observation of tau neutrino interactions.  Physics Letters B, 504(3):218–224, 2001.  10.1016/S0370-2693(01)00307-0.
  * [42] Y. Hayato & L. Pickering.  The NEUT neutrino interaction simulation program library.  The European Physical Journal Special Topics, 230:4469–4481, 2021.  10.1140/epjs/s11734-021-00287-7.
  * [43] http://t2k-experiment.org/wp-content/uploads/T2Kflux2016.tar.
  * [44] J. Allison et al.  Recent developments in Geant4.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 835:186–225, 2016.  10.1016/j.nima.2016.06.125.
  * [45] S. Agostinelli et al.  Geant4—a simulation toolkit.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 506:250–303, 2003.  10.1016/s0168-9002(03)01368-8.
  * [46] J. Allison et al.  Geant4 developments and applications.  IEEE Transactions on Nuclear Science, 53:270–278, 2006.  10.1109/tns.2006.869826.
  * [47] J. B. BIRKS.  CHAPTER 3 - THE SCINTILLATION PROCESS IN ORGANIC MATERIALS—I.  In J. B. BIRKS, editor, The Theory and Practice of Scintillation Counting, International Series of Monographs in Electronics and Instrumentation, pages 39–67. Pergamon, 1964.  10.1016/B978-0-08-010472-0.50008-2.
  * [48] A. Hiramoto et al.  First measurement of $\overline{\nu}_{\mu}$ and $\nu_{\mu}$ charged-current inclusive interactions on water using a nuclear emulsion detector.  Phys. Rev. D, 102(7):072006, 2020.  10.1103/PhysRevD.102.072006.
  * [49] H. Oshima et al.  Measurements of protons and charged pions emitted from $\nu_{\mu}$ charged-current interactions on iron at a mean neutrino energy of 1.49 GeV using a nuclear emulsion detector.  Phys. Rev. D, 106(3):032016, 2022.  10.1103/PhysRevD.106.032016.