# Evaluating Meta-Feature Selection for the Algorithm Recommendation Problem

Gean T. Pereira, Moisés R. dos Santos, André C. P. L. F. Carvalho
Computer Science Department, Institute of Mathematics and Computer Sciences
(ICMC)
University of São Paulo, São Carlos, Brazil
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

With the popularity of Machine Learning (ML) solutions, algorithms and data
have been released faster than the capacity of processing them. In this
context, the problem of Algorithm Recommendation (AR) is receiving a
significant deal of attention recently. This problem has been addressed in the
literature as a learning task, often as a Meta-Learning problem where the aim
is to recommend the best alternative for a specific dataset. For such,
datasets encoded by meta-features are explored by ML algorithms that try to
learn the mapping between meta-representations and the best technique to be
used. One of the challenges for the successful use of ML is to define which
features are the most valuable for a specific dataset since several meta-
features can be used, which increases the meta-feature dimension. This paper
presents an empirical analysis of Feature Selection and Feature Extraction in
the meta-level for the AR problem. The present study was focused on three
criteria: predictive performance, dimensionality reduction, and pipeline
runtime. As we verified, applying Dimensionality Reduction (DR) methods did
not improve predictive performances in general. However, DR solutions reduced
about 80% of the meta-features, obtaining pretty much the same performance as
the original setup but with lower runtimes. The only exception was PCA, which
presented about the same runtime as the original meta-features. Experimental
results also showed that various datasets have many non-informative meta-
features and that it is possible to obtain high predictive performance using
around 20% of the original meta-features. Therefore, due to their natural
trend for high dimensionality, DR methods should be used for Meta-Feature
Selection and Meta-Feature Extraction.

##  1 Introduction

The machine learning popularity in automated systems is increasing daily [1].
The results in the rise of machine learning applications in different areas of
interest that further expanded to meta-learning problems [2]. Meta-learning is
a sub-area of machine learning that investigates the problem of learning to
learn [3]. The aim is to develop a model that can learn new skills and rapidly
adapt to a new environment, by inspecting the relationship among problems and
performance of learning algorithms [4]. This association is further explored
for algorithm selection, which leads to the algorithm recommendation problem,
in which the most suitable algorithm is recommended according to the dataset
[5]. Specialists usually evaluate a group of algorithms for suitability to
identify the most effective recommendation algorithm for a given case. These
algorithms are trained using historical data, and their efficiency is assessed
using cross-validation methods. The best algorithm is selected based on having
the lowest predictive error or highest accuracy.

There has been a large amount of research on constructing meta-level problems
and models [6, 7, 8]. Nevertheless, few attention has been given to the
usefulness and discriminating power of dataset characteristics to identify
features that are most critical for obtaining good results [5]. To found the
most representative feature in a dataset, Feature Selection (FS) is often
employed. In this paper, we evaluate FS methods for the Algorithm
Recommendation Problem. FS is a crucial step in data mining. It is a
Dimensionality Reduction (DR) technique that decreases the number of features
to a convenient size for processing and analysis. FS is different from other
DR techniques; it does not change the original feature set; instead, it
selects a subgroup of relevant features by excluding all other features whose
existence in the dataset does not positively affect the learning model [9].

To construct a model, the only source of information used for learning is the
set of features. Thus, it is crucial to find the optimal subset of features
that represent the original dataset. However, selecting that optimal subset is
not a trivial task; if a large number of features are selected, time and space
complexity increases, causing a high workload on the classifier and a decrease
in performance [10]. On the other hand, if too few features are selected, then
there is a possibility of losing useful information, which again leads to a
decrease in model performance. Thus, an optimal subset of relevant features is
needed, which gives the best solution without decreasing the model’s
performance [11]. But there is no single feature selection method that is best
all the time. Specialists typically have to experiment with different methods
using a trial and error approach, but the problem is that this is time-
consuming and costly, especially when we have large datasets. This work aims
to present an empirical analysis of feature selection and feature extraction
in the meta-level setup for datasets related to the algorithm recommendation
problem. The present study performs three comparison criteria: predictive
performance, dimensionality reduction, and pipeline runtime.

The remaining of this work is organized as follows: Sections 2 and 3 shows the
background related to Meta-Learning for Algorithm Recommendation and Feature
Selection methods; Section 4 shows the experimental setup and methodology; The
results and proper discussion are presented in Section 5; Our conclusion and
future work directions are discussed in Section 6.

##  2 Meta-learning for Algorithm Recommendation

The work [12] was purposed the algorithm selection problem, as know as the
algorithm recommendation problem. The problem can be enunciated as the mapping
of a set of data instances to a set of candidate algorithms for optimizing a
specific performance metric. It seems intuitive that similar data sets might
have a good solution with the same algorithm. However, it is necessary to know
if exists an algorithm that has the best performance for all problem domains.
This question was explored by [13] in the "no free lunch" theorems. For
simplicity, one of the theorems demonstrates that, given a set of algorithms
applied to all possible domains, the average performance of the algorithms is
the same. This theorem for supervised machine learning implies that not exists
a unique learning algorithm with performance significant superior for all
domains.

The best solution is to test all of the algorithm candidates for a specific
task and choose to achieve the best performance. However, exists many
limitations to adopt this approach. One of the main limitations is the high
computational cost that can turn impossible to have a solution on available
time [2]. A possible solution is to use meta-learning for the algorithm
recommendation problem [2]. A contemporary definition for meta-learning or
"learn to learning" is the set of methods that consists of using the knowledge
extracted from the tasks, algorithms, or model performance evaluation of past
datasets to perform better, faster, and more efficient tasks for new datasets
[4].

Although many approaches exist for meta-learning model selection, in this
work, we focus on meta-feature-based meta-learning[3], where, given a set of
datasets and algorithm candidates, the main idea is to model the algorithm
recommendation problem as a supervised machine learning problem [14]. The
predictive attributes or meta-features are the set of characteristics
extracted from a dataset, and the target is the algorithm candidate that
achieved better performance for this dataset. Thus, the aim is to build a
model able to recommend the best algorithm for new tasks.

Meta-learning has been successfully applied to a large number of learning
tasks and domains [15] [16] [17] [7]. However, with the rising number of
purpose meta-features in the literature [18][19], it is necessary to study
feature selection methods to improve the meta-learning setup efficiency.

##  3 Feature Selection Methods

It is common sense that features used for training models have a significant
influence on the performance. With that said, irrelevant features can harm
performance. Thus, feature selection methods are employed to found the most
relevant features in a training dataset, which contribute more to the target
variable. Using feature selection can lead to less redundant data, which lower
the probability of making a decision based on noisy feature, so could reduce
overfitting and also improve the model’s accuracy. Dimensions of the training
set also decrease because feature selection chooses a relevant subset of
features within a whole dataset; thus, it also reduces the time complexity and
algorithms train faster. There are two types of feature selection methods,
model-based feature selection, and filter-based filter selection. This work
focuses on filter-based feature selection because these are methods with a low
computational cost. In the next sections, some methods for filter-based
feature selection are shortly discussed. For more information read [20] and
[21].

###  3.1 Variance Threshold

This feature selection method removes features with a low variance; thus, it
calculates each feature’s variance, then drops features with variance below
some threshold, making sure that the features have the same scale. Although it
is a simple method, it has a compelling motivation, since the idea is that low
variance features contain less relevant information. Moreover, the variance
threshold is an unsupervised method, since it only looks at the feature values
and not to the desired output.

###  3.2 Univariate Analysis

This feature selection method consists of selecting the best features based on
statistical tests. It is often used as a pre-processing step to an estimator,
thus improving the performance of a model and cutting of the time complexity,
since it reduces the dimensions of a dataset. These statistical tests are used
to found the relationship between each input feature and the output feature.
Those input feature vectors are kept and used for further analysis, which
gives a strong statistical relationship with the output variable, and
remaining input features, whose relationship is weak, are discarded. There is
much statistical test used in the univariate feature selection method i.e.,
Chi-squared, Pearson correlation, and ANOVA F-value test, and others. Two of
them are described below:

  * •

Chi-squared test: A statistical test that aims to evaluate the dependency
between two variables. Although it has some common similarities with the
coefficient of determination, the chi-square test is only valid for nominal or
categorical data, whereas the determination coefficient is only valid for
numeric data. This method has various applications, and one of them is feature
selection. Considering an input feature variable and an output variable, i.e.,
class label, first, the Chi-squared statistics of every input feature variable
concerning the output is calculated to get the dependency relationship between
them. Afterward, features that have shown less dependency with output
variables are discarded, while the features that have shown strong dependency
with the target variable are kept to training the model.

  * •

ANOVA F-test: This method is based on F-test, which is used only for
quantitative data. ANOVA F-test estimates the degree of linear dependency
between an input variable and the target variable, giving a high score to the
features which are highly correlated and low score to less correlated
features. The F-value scores inspect if the means for each group are
considerably different when we group the numerical feature by the target
vector.

##  4 Experimental Methodology

The datasets used in this work were collected from the Aslib repository [22],
being 28 meta-datasets related to the algorithm recommendation problem. These
meta-datasets are related to classic optimization problems, such as the
Traveling Salesman Problem (TSP) and the Propositional Satisfiability Problem
(SAT). Table LABEL:tab:data shows the dataset names and basic summary of them.

Table 1: Dataset’s statistics.

Dataset | Instances | Features | Classes
---|---|---|---
ASP-POTASSCO | 1294 | 139 | 11
BNSL-2016 | 1179 | 93 | 8
CPMP-2015 | 527 | 22 | 4
CSP-2010 | 2024 | 67 | 2
CSP-Minizinc-Obj-2016 | 100 | 123 | 3
CSP-Minizinc-Time-2016 | 100 | 117 | 4
CSP-MZN-2013 | 4636 | 117 | 10
GRAPHS-2015 | 5723 | 36 | 6
MAXSAT-PMS-2016 | 596 | 44 | 12
MAXSAT-WPMS-2016 | 630 | 53 | 10
MAXSAT12-PMS | 876 | 31 | 6
MAXSAT15-PMS-INDU | 601 | 57 | 16
MIP-2016 | 214 | 120 | 3
OPENML-WEKA-2017 | 105 | 157 | 3
PROTEUS-2014 | 4021 | 32 | 22
QBF-2011 | 1368 | 46 | 5
QBF-2014 | 1248 | 46 | 13
QBF-2016 | 825 | 67 | 14
SAT03-16_INDU | 2000 | 139 | 10
SAT11-HAND | 296 | 66 | 9
SAT11-INDU | 300 | 67 | 11
SAT11-RAND | 592 | 53 | 8
SAT12-ALL | 1614 | 58 | 29
SAT12-HAND | 767 | 70 | 22
SAT12-INDU | 1167 | 74 | 21
SAT12-RAND | 1362 | 98 | 9
SAT15-INDU | 300 | 87 | 10
TTP-2016 | 9714 | 58 | 18

Table 2: Classifiers and their hyperparameters.

Classifier | Hyperparameter | Default | Range
---|---|---|---
KNN | Num nearest neighboors | 3 | (1, 51)
DT | Min samples split | 2 | (2, 51)
Min samples leaf | 1 | (2, 51)
Max depth | None | (2, 31)
SVM | C | 1 | (1, 32769)
Gamma | 1/Nº features | (1, 32769)
RF | Nº estimators | 10 | (1, 1025)
Max depth | None | (1, 21)

Feature Selection methods used in this work were implemented using “scikit-
learn” [23]. For Variance Threshold, features that do not change in more than
$80$, $85$, $90$, and $95$ percent of the observations are removed. For Chi-
squared and ANOVA, features are select according to a percentile with the
highest score for each feature selection method. The percentiles considered
were $80$, $85$, $90$, and $95$.

For comparison criterion, the feature extraction technique "Principal
Component Analysis" (PCA) adds to this experiment. This technique achieved the
best results in the work of [5]. The selected components correspond to that
maintain 80, 85,90, and 95 of the original observations. These values are
stemming from [5], and the PCA implementation is available on the "scikit-
learn" package [23].

Table LABEL:tab:classifiersHyperparameters shows the classifiers used
(Classifier), such as K-Nearest Neighbors (KNN), Decision Tree (DT), Support
Vector Machine (SVM), and Random Forest (RF), their hyperparameters tuned
(Hyperparameter), the default value for the specific hyperparameter tuned
(Default) and the range values tested during the tuning of that hyperparameter
(Range). Also, in Table LABEL:tab:classifiersHyperparameters, values “None” in
DT and RF rows are the default values defined by Scikit-learn, which are said
to be expanded until all leaves are pure or until all leaves contain fewer
samples than Min samples split parameter.

For model evaluation, all the meta-datasets were scaled between 0 and 1 by
Minimum-Maximum scaler [24]. The pipeline includes feature selection methods,
meta-learners, and all parameters for the hyperparameter tuning process.
Hyperparameter tuning is made by the Random Search K-Fold cross-validation
technique in which we split our data into $k$ subsets, and then we tune and
train our models on $k-1$ folds, and the performance metric are apply on the
left fold. This process is repeated k times until all the folds have been
evaluated. Given a hyperparameter space of a specific machine learning model,
cross-validation is applied $n$ times with a random variation of
hyperparameters space. Thus, the hyperparameters set with the best performance
is selected to be applied to test data [25]. The number of $k$ subsets used
were $10$, a value widely used on the literature [26].

The models are evaluated in three criteria: predictive performance, proportion
of dimensionality reduction, and time to fit. For performance, uses the
balanced accuracy metric [27]. Balanced accuracy is a fair version of the
accuracy metric for multi-class problems. The proportion of dimensionality
reduction is the difference between the number of dimensions of the original
meta-dataset and the dimensions after feature selection or extraction, divided
by the number of dimensions of the original meta-dataset. Time to fit is the
average time to induce the feature selection or extraction and the
classification model. This time is returned by the random search cross-
validation function of the “scikit-learn” package. For reproducibility, all
the paper experiment’s implementation is public on this repository:
https://github.com/moisesrsantos/featselec.

##  5 Results and Discussion

This section discuss the results of applying 4 meta-learners (SVM, KNN,
Decision Tree, and Random Forest) to 28 meta-datasets related to algorithm
recommendation, which were also submitted to 3 Feature Selection methods (Chi-
squared, F-ANOVA and Variance threshold) and 1 Feature Extraction method
(PCA). Our experiments were performed to answer three research questions: (1)
Is there any predictive performance improvement of applying feature selection
to meta-datasets related to algorithm recommendation? (2) How good is feature
selection to reduce the dimensionality of those meta-datasets? (3) Is there
any impact on the pipeline execution time when using feature selection?

The following results are divided in three parts, each one aiming to answer
one of the research questions. We used bar plots and box-plots to discuss the
properties of the results distributions, as well as Critical Difference (CD)
diagrams to better show the statistical results between comparisons. In
Figures 2, 4, and 6, the diagrams illustrate the Friedman-Nemenyi test with
95% of confidence for the Balanced Accuracy, percentage of dimensionality
reduction, and pipeline execution times. In the CD diagrams, the more towards
to 1, the better is the method. Therefore, the methods on the left have better
performances than the methods on the right. Besides, it is important to
consider the CD intervals that cross the lines representing the methods. These
intervals demonstrate that all the lines that touch it have statistically
equal values, so no method is statistically better than the others and, thus,
they all have comparable performances.

###  5.1 Predictive Performance

In order to verify whether the DR methods bring some improvement in terms of
predictive performance, we analyzed the distributions of balanced accuracy
obtained by applying each DR method to each meta-dataset and training with
each meta-learner. As shown by box-plots in Figure 2, all DR methods had
similar performances, even when compared to the original datasets. Slightly
standing out from the others, the Variance threshold appeared to reduce the
variance of the predictive performance across datasets and classifiers,
showing a lower range of values. However, it also generated more sparse
maximum outliers. Speaking of outliers, another interesting thing to observe
is that the number of outliers remains pretty much the same regardless of the
DR method applied, and those outliers remain located in the region of the
maximum values only.

Figure 1: Balanced Accuracy comparison between dimensionality reduction
methods.

CD = 0.5812345OriginalF ANOVAChi-squaredVariancePCA

Figure 2: Critical Difference diagram of the performances.

Even though Variance showed more stable results (except for its outliers),
when submitted to a statistical test, it was verified that F-ANOVA had the
best performance between the DR methods. However, considering the CD interval
shown in Figure 2, no DR method improved the overall predictive performance in
the task of algorithm recommendation with statistical significance. Even so,
using DR was as good as using the original data.

###  5.2 Dimensionality Reduction

For the purpose of verifying how good are the feature selection methods in
reducing the dimensionality of the selected meta-datasets, we focused on
observing the performance of each technique in a broader context. Therefore,
reductions in all pipelines involving each meta-dataset and all meta-learners
were considered. As seen in Figure 4, DR methods showed a high percentage of
reduction in average, with all of them (except for Variance) achieving more
than 80% percent reduction. Chi-squared, F-ANOVA, and PCA presented similar
reduction averages in a stable manner. In turn, Variance proved to be inferior
in average and more unstable, showing higher standard deviation values.

Figure 3: Percentage of reduction.

CD = 0.441234F ANOVAPCAVarianceChi-squared

Figure 4: Critical Difference diagram of the dimensionality reduction.

When considering the statistical comparisons seen in Figure 4, it is possible
to notice that F-ANOVA, PCA, and Chi-squared did not differ statistically,
being equally good in reducing the dimensions of the original meta-data.
Furthermore, it is clear that Variance is statistically worse than others.

###  5.3 Pipeline Runtime

To verify if there is any impact on the pipeline runtimes when using feature
selection, we collected runtimes from all the pipelines performed in our
experiments. Those runtimes included both the time to extract the sub-set of
meta-features (feature selection) or transform the original space of meta-
features (feature extraction) and the runtime to fit the data. The
distributions of the runtimes for each DR method and the original setup are
shown in Figure 6. As seen, the times of the pipelines were short in general,
with maximum values achieving close to 2 seconds. The medians for these
distributions were similar and much closer to the Q1 and the minimum. It is
interesting to see that the variations happened mainly in the region of the
maximum values, while the minimum values, Q1, and Interquartile range were
similar.

Figure 5: Pipeline runtime comparison.

CD = 0.5812345VarianceChi-squaredPCAOriginalF ANOVA

Figure 6: Critical Difference diagram regarding runtimes.

In summary, these results show that the pipelines were cheap, and the
application of DR methods does not seem to have much impact on time. However,
when considering the CD diagram shown in Figure 6, it is possible to see that
the execution times for Variance, Chi-square, and F-ANOVA, are statistically
comparable and shorter than using the original data and PCA. Using Feature
Selection proved to be a better solution for reducing the overall time of
pipelines than not using it, showing that, despite the additional time to
lower dimensions, the total time, including the training, is shorter. On the
other hand, using or not using PCA did not statistically change the runtime of
the pipelines.

##  6 Conclusion and Future Works

We presented a comparative study of Feature Selection and Feature Extraction
in the meta-level setup for datasets related to Algorithm Recommendation. Our
goal was to evaluate popular DR methods according to three criteria:
predictive performance, dimensionality reduction, and pipeline runtime. As we
verified, applying DR did not improve predictive performance in general.
However, it reduced around 80% of the meta-features yet achieving the same
performance as the original setup with a shorter runtime. The only exception
was PCA, which showed about the same runtime. Our experiments showed that
these meta-datasets have many non-informative meta-features and that it is
possible to achieve good performance using about 20% of the original features.
Traditional Meta-Learning, the one focused on hand-craft extraction of meta-
features, seems to suffer a lot with the curse of dimensionality. Instead of
helping Machine Learning algorithms to achieve better or fast performance
while taking advantage of high-level information (meta-level), it seems that,
in some cases, meta-info hinders the task. We conclude that using DR methods
is beneficial to many meta-datasets since their natural trend for high
dimensionality. However, we acknowledge that more experiments with different
numbers and types of meta-features are needed to ensure our claim.

## References

  * [1] Michael I Jordan and Tom M Mitchell.  Machine learning: Trends, perspectives, and prospects.  Science, 349(6245):255–260, 2015.
  * [2] Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta.  Metalearning: Applications to data mining.  Springer Science & Business Media, 2008.
  * [3] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren.  Automatic machine learning: methods, systems, challenges.  Challenges in Machine Learning, 2019.
  * [4] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org, 2017.
  * [5] Edesio Alcobaça, Rafael G Mantovani, André LD Rossi, and André CPLF de Carvalho.  Dimensionality reduction for the algorithm recommendation problem.  In 2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pages 318–323. IEEE, 2018.
  * [6] Gean Trindade Pereira, Moisés dos Santos, Edesio Alcobaça, Rafael Mantovani, and André Carvalho.  Transfer learning for algorithm recommendation.  arXiv preprint arXiv:1910.07012, 2019.
  * [7] Gabriel Jonas Aguiar, Rafael Gomes Mantovani, Saulo M Mastelini, André CPFL de Carvalho, Gabriel FC Campos, and Sylvio Barbon Junior.  A meta-learning approach for selecting image segmentation algorithm.  Pattern Recognition Letters, 128:480–487, 2019.
  * [8] Gabriel Fillipe Centini Campos, Saulo Martiello Mastelini, Gabriel Jonas Aguiar, Rafael Gomes Mantovani, Leonimer Flávio de Melo, and Sylvio Barbon.  Machine learning hyperparameter selection for contrast limited adaptive histogram equalization.  EURASIP Journal on Image and Video Processing, 2019(1):59, 2019\.
  * [9] Isabelle Guyon, Steve Gunn, Masoud Nikravesh, and Lofti A Zadeh.  Feature extraction: foundations and applications, volume 207.  Springer, 2008.
  * [10] Peter Flach.  Machine learning: the art and science of algorithms that make sense of data.  Cambridge University Press, 2012.
  * [11] Stephen Marsland.  Machine learning: an algorithmic perspective.  CRC press, 2015.
  * [12] John R Rice.  The algorithm selection problem.  In Advances in computers, volume 15, pages 65–118. Elsevier, 1976\.
  * [13] David H Wolpert.  The lack of a priori distinctions between learning algorithms.  Neural computation, 8(7):1341–1390, 1996.
  * [14] Ricardo Vilalta and Youssef Drissi.  A perspective view and survey of meta-learning.  Artificial intelligence review, 18(2):77–95, 2002.
  * [15] Jorge Y Kanda, Andre CPLF de Carvalho, Eduardo R Hruschka, and Carlos Soares.  Using meta-learning to recommend meta-heuristics for the traveling salesman problem.  In 2011 10th International Conference on Machine Learning and Applications and Workshops, volume 1, pages 346–351. IEEE, 2011.
  * [16] Antonio Rafael Sabino Parmezan, Huei Diana Lee, and Feng Chung Wu.  Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework.  Expert Systems with Applications, 75:1–24, 2017.
  * [17] Thiyanga S Talagala, Rob J Hyndman, George Athanasopoulos, et al.  Meta-learning how to forecast time series.  Monash Econometrics and Business Statistics Working Papers, 6:18, 2018.
  * [18] Adriano Rivolli, Luís PF Garcia, Carlos Soares, Joaquin Vanschoren, and André CPLF de Carvalho.  Towards reproducible empirical research in meta-learning.  arXiv preprint arXiv:1808.10406, 2018.
  * [19] Adriano Rivolli, Luís PF Garcia, Carlos Soares, Joaquin Vanschoren, and André CPLF de Carvalho.  Characterizing classification datasets: a study of meta-features for meta-learning.  arXiv preprint arXiv:1808.10406, 2018.
  * [20] Cosmin Lazar, Jonatan Taminau, Stijn Meganck, David Steenhoff, Alain Coletta, Colin Molter, Virginie de Schaetzen, Robin Duque, Hugues Bersini, and Ann Nowe.  A survey on filter techniques for feature selection in gene expression microarray analysis.  IEEE/ACM Transactions on Computational Biology and Bioinformatics, 9(4):1106–1119, 2012.
  * [21] Andrea Bommert, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang.  Benchmark for filter methods for feature selection in high-dimensional classification data.  Computational Statistics & Data Analysis, 143:106839, 2020.
  * [22] Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Fréchette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, et al.  Aslib: A benchmark library for algorithm selection.  Artificial Intelligence, 237:41–58, 2016.
  * [23] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [24] Jiawei Han, Jian Pei, and Micheline Kamber.  Data mining: concepts and techniques.  Elsevier, 2011.
  * [25] James Bergstra and Yoshua Bengio.  Random search for hyper-parameter optimization.  Journal of machine learning research, 13(Feb):281–305, 2012.
  * [26] Stuart J Russell and Peter Norvig.  Artificial Intelligence: A Modern Approach.  Prentice Hall, 2009.
  * [27] Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann.  The balanced accuracy and its posterior distribution.  In 2010 20th International Conference on Pattern Recognition, pages 3121–3124. IEEE, 2010.


11institutetext: KVI-CART, University of Groningen, Groningen, The Netherlands
22institutetext: Department of Physics, University of Sistan and Baluchestan,
Zahedan, Iran 33institutetext: Institute of Theoretical Physics and Astronomy,
Vilnius University, Saulėtekio al. 3, LT-10222 Vilnius, Lithuania
44institutetext: Department of Physics, School of Science, Yazd University,
Yazd, Iran 55institutetext: M. Smoluchowski Institute of Physics, Jagiellonian
University, PL-30348 Kraków, Poland 66institutetext: Institute of Nuclear
Physics, PAS, PL-31342, Kraków, Poland 77institutetext: Department of Physics,
Faculty of Science, University of Kashan, Kashan, Iran 88institutetext:
Department of Physics, University of Tehran, Tehran, Iran 99institutetext:
Institute of Physics, University of Silesia, Chorzów, Poland

# Analyzing powers in $d(\mathaccent 382{p},pp)n$ at intermediate and large
scattering angles at 190 MeV

M. Mohammadi-Dadkan e-mail<EMAIL_ADDRESS>H.R. Amir-Ahmadi 11
M.T. Bayat 11 A. Deltuva 33 M. Eslami-Kalantari 44 J. Golak 55 N. Kalantar-
Nayestanaki 11 St. Kistryn 55 A. Kozela 66 H. Mardanpour 11 A.A. Mehmandoost-
Khajeh-dad e-mail<EMAIL_ADDRESS>J.G. Messchendorp e-mail:
<EMAIL_ADDRESS>A. Ramazani-Moghaddam-Arani 77 R. Ramazani-
Sharifabadi 1188 R. Skibiński 55 E. Stephan 99 H. Tavakoli-Zaniani 1144 H.
Witała 55

(Received: date / Revised version: date)

###### Abstract

Understanding of the exact nature of three-nucleon forces is the most
challenging topic in the field of nuclear physics. Three-nucleon break-up
reaction is a good tool to look into the underlying dynamics of the nuclear
force, thanks to its rich kinematical phase space which has different levels
of sensitivity to three-nucleon force effects. The recent studies on few-
nucleon systems have revealed that the current nuclear force models cannot
describe nucleon-deuteron scattering data accurately. In the present work, the
analyzing powers of the proton-deuteron break-up reaction obtained using a
$190$ MeV polarized proton beam will be discussed. We present for the first
time the vector analyzing powers for the kinematics in which one of the
protons scatters to intermediate and large scattering angles at this energy.
The results show a fairly good agreement with various theoretical predictions
for both intermediate and large scattering angles of the break-up phase space.

###### pacs:

21.45.+vFew-body systems and 13.75.CsNucleon-nucleon interactions

††offprints:

##  1 Introduction

Since 1932 when Chadwick discovered the neutron as a constituent of atomic
nuclei Chadwick:1932 , efforts have been made to build a framework for
describing the interaction between nucleons. Yukawa initiated the first
systematic approach for describing the nucleon-nucleon (NN) interaction in
analogy to the electromagnetic interaction Yukawa:1935 . His formulation of
the nuclear force was based on the idea that the proton and neutron are
fundamental particles. Therefore, the potentials based on these models solely
take the nucleons and mesons as degrees of freedom in the nuclei. Different NN
potentials have been developed in the past decades. They usually carry the
name of the group which developed them such as AV18 Wiringa1995 , CD-Bonn
Machleidt2001 , etc.. Each of these models has different parameters and they
are fitted to the empirical pp and np scattering database. While these models
do a great job in describing two-nucleon systems below the pion-production
threshold energy, they fail to describe the systems which have more than two
nucleons. There are lots of evidences for this by comparing data and theory
for both the nuclear binding energies Pieper2015 ; Noga2003 ; Noga2000  and
various scattering observables Sakai2000 ; Ermisch2001 ; Cadman2001 ;
Hatanaka2002 . It has became clear that there are additional underlying
dynamics, beyond the NN interaction, playing a role in the nuclear force and
which are referred to as many-body force effects. We expect that the three-
nucleon force (3NF) is the dominant part in the hierarchy of many-body force
effects.

Figure 1: The left panel illustrates the structure and the components of BINA
(Big Instrument for Nuclear-polarization Analysis) together with the angular
coverage of each part. The right panel shows the partly-assembled backward-
ball (top figure) and the definition of the angles of typical detectors
(bottom figure). The centroid of each ball detector is considered as the
angular position of that detector; see the black points in the bottom-right
figure.

In the past decades, several phenomenological 3NF models such as Urbana-
Illinois (UIX) URB1983 ; URB1997  and CD-Bonn+$\Delta$ Nemoto1998 ;
Deltuva20032 ; Deltuva20033  have been developed based on the works by Fujita
and Miyazawa FujitaMiyazawa  and Brown and Green BROWN1969  to implement 3NF
effects in the interaction of nucleons. However, different studies have shown
that the inclusion of current models of 3NF do not completely fill the gap
between data and calculations. For a detailed discussion of the progress of
the field at intermediate energies, see Refs. Kalantar2011 ; Kistryn2013 .

Ever since it was known that nucleons are composed of quarks and gluons, the
nuclear force has been considered as residual color force between quarks and
gluon. Due to the non-perturbative nature of QCD at low energies, it can, with
the present available techniques, only be applied to the nuclear system
through an Effective Field Theory (EFT). This fundamental approach for
nucleonic systems is known as Chiral Perturbation Theory (ChPT) WEINBERG1990 .
To take into account all the effective elements in ChPT calculations, one
should extend the calculations to higher orders. The work toward higher orders
of ChPT is still in progress. A detailed overview of the present state-of-the-
art of the ChPT approach can be found in Refs. Kolck1994 ; Kolck1999 ;
MACHLEIDT20111 ; Epelbaum2019 ; Binder2018 . Presently, there are no results
available from the ChPT calculations for comparison with the data shown in
this paper. For a proton beam energy of $190$ MeV, higher orders should be
included in the calculations before a fair comparison can be made.

In the past decades, the elastic channel in Nd scattering has been
investigated experimentally at different beam energies below the pion-
production threshold. The results showed that the current 3NF models are not
able to describe the discrepancy between measurements and calculations
particularly at higher beam energies and at the minimum of the differential
cross section as well as at backward angles Hatanaka2002 ; Ermisch2005 .
Moreover, the inclusion of 3NFs into 3N Faddeev calculation has failed to
describe the analyzing power $A_{y}$ at low energies, an effect known in
literature as $A_{y}$-puzzle AyPuzzle1998 ; Aypuzzle2001 . The elastic channel
has a limited kinematical phase space and one cannot test all the aspects of
the theoretical models. To have a systematic and a detailed investigation of
3NFs, the three-body break-up channel is a suitable candidate because of its
rich kinematical phase space yielding various degrees of sensitivity to the
underlying nucleon-nucleon and three-nucleon forces.

A systematic investigation of 3NFs through elastic and the break-up reaction
has been initiated at KVI in a common effort between the Dutch and the Polish
groups since the end of the 90s by developing various experimental setups and
exploiting high-quality polarized beams Ermisch2005 ; KistrynSALAD2005 ;
KISTRYNCoulomb2006 ; Stephan2007 ; Amirahmadi2007 ; Mehmandoost2011 . BINA
(Big Instrument for Nuclear-polarization Analysis) is the latest experimental
setup which was exploited at KVI for few-nucleon scattering experiments. This
detection system is capable of measuring the energy and the scattering angles
of all the reaction yields of three and four-body final states in coincidence.
A series of experiments has been conducted using BINA to study the deuteron
break-up channel using polarized proton and deuteron beams in a wide range of
energies between 50 and $190$ MeV/nucleon Mardanpour_thesis ; Eslami_thesis ;
RamazaniMoghadam_thesis ; BINA80mev ; BINA100 . In this article, we present
the measurements of the analyzing powers of the proton-deuteron break-up
reaction obtained with a proton-beam energy of $190$ MeV. A specific part of
the phase space of this experiment has already been published MARDANPOUR2010 .
In this work, we focused on the break-up kinematics in which one proton
scatters to forward angles ($\theta_{1}<32^{\circ}$) and the other to backward
angles ($\theta_{2}>40^{\circ}$). The experimental setup will be described in
Sec. 2. Then, the analysis method and the results will be presented.

##  2 Experimental setup

The present experiment was conducted utilizing the AGOR (Accélérateur
Groningen-ORsay) cyclotron at KVI delivering a high-quality polarized-proton
beam of $190$ MeV. For details about the experimental setup, see Ref.
Mardanpour_thesis . BINA, is a specially designed 4$\pi$ detection system with
which 3NF effects could be studied through the three-body and four-body
elastic as well as break-up reactions. The detection system consists of
different components which make it capable of measuring energies up to $140$
MeV per nucleon, scattering angles between $10^{\circ}$-$160^{\circ}$, and
almost a full coverage of azimuthal angles. There are two features that make
BINA unique among other detection systems in the field of few-nucleon
scattering:

  * •

Detection of all the charged particles in the final-state of the three-body
and the four-body reactions in coincidence;

  * •

The almost 4$\pi$ angular coverage which can probe a large part of the phase
space of the break-up reactions.

In general, BINA has two main parts, the forward-wall and the backward-ball.
The left part of Fig. 1 illustrates different parts of BINA. The forward-wall
consists of a cylindrically-shaped array of scintillators ($E$-detectors) to
measure the energy of charged particles, a MWPC to measure their scattering
angles, and a vertical array of $\Delta E$-detectors that is used in
combination with $E$-detectors for particle identification. The forward-wall
covers angles which are in the range of $10^{\circ}<\theta<35^{\circ}$. The
backward-ball is a ball-shape scattering chamber which consists of 149
pyramid-shaped plastic scintillators. The geometrical design of the backward-
ball and its building blocks are similar to the classic soccer ball. The
position of each backward-ball scintillator is labeled with a specific
$\theta$ and $\phi$ in the laboratory coordinate system. The backward-ball can
measure the scattering angles of the particles with a precision of $\pm
10^{\circ}$. The right panels of Fig. 1 demonstrate the structure of the
backward-ball and the positioning of the detector elements. This part of BINA
is capable of measuring the energy and the scattering angles of the particles
in the range of $35^{\circ}<\theta<160^{\circ}$ except where the target holder
is attached to the scattering chamber. Therefore, there are no detectors at
the part of the phase space with polar and azimuthal angles in the range of
$80^{\circ}<\theta<120^{\circ}$ and $70^{\circ}<\phi<110^{\circ}$,
respectively. Also, at the angular range of $80^{\circ}<\theta<100^{\circ}$,
the scattered protons partially lose their energies in the target-holder ring.
Due to the complicated procedure for the energy correction for these protons,
we decided to leave out these detectors from the analysis. The detectors which
are located at $\theta>140^{\circ}$ are excluded from this analysis as well
because the expected energy of the break-up protons is lower than the
threshold of these detectors. The parts that were used in the analysis cover
scattering angles within $40^{\circ}<\theta<80^{\circ}$ and
$100^{\circ}<\theta<140^{\circ}$. In the present analysis, we obtained the
analyzing powers of the proton-deuteron break-up reaction for the kinematics
in which one proton was registered in the forward-wall and the other one in
the backward-ball.

##  3 Analysis method

In the proton-deuteron break-up reaction, there are three particles in the
final state, two protons and one neutron. From the kinematic point of view,
there are 9 variables $(\theta_{i},\phi_{i},E_{i})$ for this reaction channel
which can be measured. Applying energy and momenta conservation reduces the
number of degrees of freedom to 5. It is, therefore, sufficient to measure 5
of the 9 variables to have all the information of the reaction. BINA is able
to measure the energy and the scattering angles of two protons in coincidence
which provides an extra redundancy of one degree of freedom. Conventionally,
in the three-body break-up reaction, each kinematical configuration is defined
with $(\theta_{1},\theta_{2},\phi_{12}=|\phi_{2}-\phi_{1}|)$ which represents
the polar scattering angles of protons 1 and 2 and the relative azimuthal
angle between them. Figure 2 shows the definition of the scattering angles of
two protons in the $p$-$d$ break-up reaction. There are two conventions for
defining the $y$-axis OHLSEN1981 . In this paper, we use the asymmetric choice
for the azimuthal angles, where $\vec{k}_{1}$ lies in the $xz$ plane, and
therefore $\phi_{1}=0$, see Fig. 2. Here, $\vec{k}_{1}$ is the momentum of the
outgoing proton which scattered to smaller polar angle $\theta$. For each
kinematical configuration, the energy correlation of the two protons is
referred to as the $S$-curve. By employing the $S$-curve, a variable $S$ is
defined as an arc length of the curve with a starting point at the minimum
energy of one of the protons. In the present work, we used a convention in
which the starting point of $S$ is the minimum value of $E_{2}$ and it
increases counter-clockwise as it is shown in Fig. 3. Here, $E_{1}$ and
$E_{2}$ correspond to $\vec{k}_{1}$ and $\vec{k}_{2}$, respectively.
Traditionally, the three-body break-up observables are presented as a function
of $S$-value.

Figure 2: The scattering diagram of a three-body final-state reaction.
$\mathaccent 382{k}$1 and $\mathaccent 382{k}_{2}$ are the momentum of the two
detected particles with scattering angels $\theta_{1}$ and $\theta_{2}$,
respectively. $\phi_{2}$ is the angle between the projection of $\mathaccent
382{k}_{2}$ on the $x-y$ plane and positive direction of the $x$-axis.
$\hat{s}$ denotes the direction of spin of the projectile. $\phi$ is the angle
between the projection of $\hat{s}$ on the $x-y$ plane and the $y$-axis.
$\beta$ is the angle between spin of the projectile and its momentum direction
($z$-axis). In the present experiment, the polarization of beam is
perpendicular to the direction of the beam momentum and, therefore
$\beta=90^{\circ}$.

Figure 3: The kinematically allowed relation of the energies of the two
protons of the three-body break-up reaction, so called $S$-curve, for
($\theta_{1}=25^{\circ},\theta_{2}=45^{\circ},\phi_{12}=160^{\circ}$) along
with the experimental data. In this analysis, we took the minimum of $E_{2}$
as the starting point of $S$-value. The protons which have energies
$E_{1}>140$ MeV punch through the E-detectors of the forward-wall and deposit
part of their energies in the detectors resulting in a kink in the spectrum.
The dashed lines illustrate the definition of a typical $S$-bin for $S=127$
MeV. The events within each $S$-bin were used to extract the asymmetry
distributions and the analyzing powers.

To investigate spin effects in the nuclear force, one should measure various
spin-related observables such as analyzing powers. The polarized beam (or
target) imposes an asymmetry on the cross section of the break-up reaction. In
this article, we present a study of the vector analyzing powers, $A_{x}$ and
$A_{y}$, in the proton-deuteron break-up reaction. $A_{x}$ is odd and $A_{y}$
is even under the parity inversion. Here, the parity inversion corresponds to
changing the sign of the azimuthal scattering angle of the two outgoing
protons. By exploiting the formulation taken from Ref. OHLSEN1981  for
spin-$\frac{1}{2}$ protons, the following expressions can be derived for two
parity modes:

$\Upsilon^{+}(\xi,\phi)=\frac{N_{p}^{\uparrow}-N_{p}^{\downarrow}}{N_{p}^{\downarrow}p_{z}^{\uparrow}-N_{p}^{\uparrow}p_{z}^{\downarrow}}=A_{y}\cos\phi-
A_{x}\sin\phi,$ (1)
$\Upsilon^{-}(\xi,\phi)=\frac{N_{m}^{\uparrow}-N_{m}^{\downarrow}}{N_{m}^{\downarrow}p_{z}^{\uparrow}-N_{m}^{\uparrow}p_{z}^{\downarrow}}=A_{y}\cos\phi+A_{x}\sin\phi.$
(2)

where $\Upsilon^{+}(\xi,\phi)$ and $\Upsilon^{-}(\xi,\phi)$ are the asymmetry
terms for two parity modes and $\xi$ represents any appropriate set of
kinematical variables. $N_{p}^{\uparrow}$ ($N_{p}^{\downarrow}$) are the
number of events for up (down) polarization modes of the ($\phi_{1},\phi_{2}$)
setup. Similarly, $N_{m}^{\uparrow}$ ($N_{m}^{\downarrow}$) are the number of
events for up (down) polarization modes of the ($-\phi_{1},-\phi_{2}$) setup.
$p_{z}^{\uparrow}$ and $p_{z}^{\downarrow}$ are the polarization degrees of up
and down modes, respectively. $A_{x}$ and $A_{y}$ are the two components of
the vector analyzing power and $\phi$ is the azimuthal angle of the reaction
plane. Two asymmetry components can be constructed from $\Upsilon^{+}(\phi)$
and $\Upsilon^{-}(\phi)$ to extract $A_{x}$ and $A_{y}$ independently:

$g(\xi,\phi)=\frac{\Upsilon^{-}(\xi,\phi)-\Upsilon^{+}(\xi,\phi)}{2}=A_{x}\sin\phi,$
(3)
$h(\xi,\phi)=\frac{\Upsilon^{-}(\xi,\phi)+\Upsilon^{+}(\xi,\phi)}{2}=A_{y}\cos\phi.$
(4)

Figure 4: The reconstructed neutron missing-mass for one of the $S$-bins. The
peak position of the spectrum which is around the expected missing-mass of
neutron indicates that the detectors are well calibrated. The interval of $\pm
3\sigma$ was used to obtain the number of break-up events under the peak after
subtracting the background. The background was estimated using the sensitive
non-linear iterative peak clipping algorithm (red line). The green histogram
shows the missing-mass spectrum after background subtraction. The vertical
black dash lines illustrate the integration window.

After the energy calibration of the forward-wall detectors Mardanpour_thesis ,
each of the backward-ball detectors was calibrated using the energy
information of a break-up proton (1) registered in the forward-wall in
coincidence with the other proton (2) arriving at one of the backward-ball
detectors. Further, the protons with energies larger than $140$ MeV
($E_{1}>140$ MeV) punch through the $E$-detectors of the forward-wall and
consequently deposit part of their energies in the detectors, see Fig. 3. The
points corresponding to the region of punch through are excluded in the final
spectrum. Events were selected for each kinematic of ($\theta_{1}\pm
2.5^{\circ},\theta_{2}\pm 10^{\circ},\phi_{12}\pm 10^{\circ}$). The $S$-curve
is divided into 15 equally-sized bins and the events within each $S$-bin are
used to reconstruct the missing-mass of the neutron. Figure 3 shows the
selected events for configuration of
$(\theta_{1},\theta_{2},\phi_{12})=(25^{\circ},45^{\circ},160^{\circ})$
together with the corresponding $S$-curve. The number of counts was obtained
for each spin state and parity mode using the corresponding missing-mass
spectrum after subtracting the background, see Fig. 4. The background mainly
stems from uncorrelated protons from two different reactions, either
elastically scattered or from a break-up reaction, and accepted by the
hardware trigger as candidate break-up events. Figure 4 shows the missing-mass
spectrum for the $S$-bin shown in Fig. 3. The cut on the $S$-curve induces an
artificial correlation between the two time-uncorrelated hits that mainly stem
from two protons of the elastic proton-deuteron scattering process and of the
break-up reaction. As a consequence, a peaking structure of the background is
observed in the missing-mass spectrum for the selected $S$-bin. The location
of this peak depends upon $S$ and is well understood. To account for the
background including the one from the elastic channel, we used an algorithm
which was developed to estimate background of gamma-rays spectrum MORHAC1997 .
This method allows to separate continuous background from peaks, based on a
sensitive non-linear iterative peak clipping algorithm. The algorithm was
applied to each neutron missing-mass spectrum to estimate the background shape
using an $8^{\rm th}$-order polynomial function. The main motivation for
employing this method was the complicated shape of the background for each
detector and spin state. Figure 4 shows the estimated background shape for a
typical missing-mass spectrum. To find the peak position of the missing-mass
spectrum, we used an algorithm for peak searching MORHAC2000 . It allows to
automatically identify the peaks in a spectrum in the presence of the
continuous background and statistical fluctuations. The algorithm is based on
smoothed second differences that are compared to its standard deviations.
Therefore, it is necessary to pass a value for the width of the peak
($\sigma$) to the peak searching function. The algorithm is selective to the
peaks with a given $\sigma$. Here, we fixed $\sigma$ to $7$ MeV/$c^{2}$ which
is equal to the expected width of the peak in the missing-mass spectrum. Also,
to have a consistent counting of events in the four missing-mass spectra, spin
up and down and two parity modes, the peak position of one of the spectra was
used for the other spectra. The number of counts was obtained by including the
events which were located $\pm 3\sigma$ of the peak position. To extract the
analyzing powers from Eqs. 3 and 4, the number of counts for each polarization
and parity modes are extracted and normalized with the integrated charge from
the Faraday cup. The $g(\xi,\phi)$ and $h(\xi,\phi)$ are constructed using the
normalized counts as a function of $\phi$. To measure the values of analyzing
powers, the functions $A\sin\phi+B$ and $C\cos\phi+D$ are fitted to
$g(\xi,\phi)$ and $h(\xi,\phi)$, respectively. The parameters $A$ and $C$
represent the analyzing powers $A_{x}$ and $A_{y}$, respectively. The
parameters $B$ and $D$ are free parameters to correct for a possible offset of
the charge measurement and other normalization factors which we missed during
the analysis. Figure 5 shows the results of the fits for $g(\xi,\phi)$ and
$h(\xi,\phi)$ for one of the analyzed configurations. The quality of the fits
for all configurations has been found to be good and an average
$\chi^{2}/$ndf$\leavevmode\nobreak\ \sim 1$ was obtained.

Figure 5: The constructed asymmetry distributions for $A_{x}$ (top panel) and
$A_{y}$ (bottom panel) for the kinematics
($25^{\circ},45^{\circ},160^{\circ}$) and $S=76$ MeV. The quality of fit and
the obtained values of the analyzing powers are shown at the top left of each
panel. The statistical errors are obtained using the number of counts before
subtracting the background.

##  4 Results and discussion

The proton-deuteron break-up reaction has a rich kinematical phase space. In
the present work, we measured the vector analyzing powers for the kinematics
in which one proton scatters to forward angles ($\theta_{1}<32^{\circ}$) and
the other to intermediate ($\theta_{2}=45^{\circ}$) and large
($\theta_{2}=134^{\circ}$) scattering angles. Due to the low angular
resolution in the backward ball, all the theoretical calculations have been
averaged over the experimental angular bins. For the averaging, we used the
theoretical cross sections and solid angles as weighting factors. For each
non-coplanar (coplanar) kinematic, we exploited 125 (75) sub-configurations
with a $1^{\circ}$ step in $\theta_{1}$, $4^{\circ}$ step in $\theta_{2}$, and
$2^{\circ}$ step in $\phi_{12}$. Figure 6 shows a comparison between the
results of the averaging and the central values for some configurations. Even
though the averaging does not show a significant deviation from the
calculation based on the central value of the angle of the bin for most
kinematics, we performed this procedure for all of them to have a consistent
comparison between data and theory.

Figure 6: Comparison of the theoretical predictions (AV18) at the center of
the angular bins (red line) and averaged over the angular bins (black line)
for some kinematical configurations. The gray lines are the sub-configurations
which are within the detector acceptance and are used to perform averaging.

The measured analyzing powers are presented in Figs. 7-10 for a set of
configurations at intermediate ($\theta_{2}=45^{\circ}$) and large
($\theta_{2}=134^{\circ}$) scattering angles, and compared with Faddeev
calculations using various NN and NN+3NF potentials. The red dash and solid
lines represent the predictions of CD-Bonn and CD-Bonn+$\Delta$ Deltuva20032 ;
Deltuva20031 ; Deltuva20033 , respectively. The blue dash and solid lines are
the predictions of AV18 and AV18+UIX GLOCKLE1996 ; Witala2002 ;
WiatalaComp2002 , respectively. The results of all the calculations are
averaged over the experimental angular bins. The Coulomb effect is negligible
in the analyzing powers MARDANPOUR2010 . Therefore, the results of CD-
Bonn+$\Delta$+Coulomb are not included for ease of comparison with the other
calculation. The cyan band in each figure represents the systematic
uncertainty which consists of three parts. The polarization of the beam was
measured by analyzing the elastic proton-deuteron channel with BINA and by
using the measurements of the In-Beam Polarimeter (IBP). Typically, the beam
polarization varied gradually between $0.5$-$0.7$ during the course of the
experiment. The associated uncertainties of the polarization values are $6\%$
statistical and $3\%$ systematical Mardanpour_thesis . This uncertainty in the
polarization causes a systematic error of $\sigma_{pol}\sim 7\%$ in the
analyzing power measurement. Another source of systematic uncertainty is
identified to be from residual and unknown asymmetries caused by efficiency
variations between the up and down polarization states of the beam, error in
charge normalization, etc.. All of these will result in a “wrong” model
presented in Eqs. 3 and 4. This source of systematic uncertainty was
investigated through comparing the results with and without the free
parameters of the fitting functions on the asymmetry distributions. We
obtained a systematic uncertainty $\sigma_{asymm}\sim 0$-$0.1$ for $A_{x}$ and
$\sigma_{asymm}\sim 0$-$0.12$ for $A_{y}$. The third source of the systematic
uncertainty comes from the shape of the background. This part of the
systematic uncertainty consists of two components. The first one is the
difference between results of full and limited ranges of the background
estimation, see Fig. 4. To do this, we keep the polynomial order to be the
same but decrease the range of the background estimation. The error from this
part was estimated to be in the range of $\sigma_{bg1}\sim 0$-$0.04$ for both
$A_{x}$ and $A_{y}$. The other component of the error in the shape of the
background was obtained using the difference between the results of the
background estimation using two different polynomial orders (6 and 8) but
keeping the full range. The error from this source was estimated to be in the
range of $\sigma_{bg2}\sim 0$-$0.04$. The total systematic uncertainty was
obtained by a quadratic sum of the four sources of systematic errors, assuming
them to be independent. The total systematic uncertainty varies between
$0$-$0.16$ for $A_{x}$ and $0$-$0.2$ for $A_{y}$ depending on the
configuration. We decided only to present data for which the total systematic
uncertainty is less than $0.08$ to have a meaningful comparison with the
calculations. This operation only excluded a small fraction of the data
points.

Figures 7-10 show the results for $\theta_{2}=45^{\circ}$ and $134^{\circ}$
and different combinations of $\theta_{1}$ and $\phi_{12}$. For some of the
configurations where $\theta_{2}=45^{\circ}$, the data points at $S>100$ MeV
are not presented because we were not able to subtract the background of the
uncorrelated elastic break-up events unambiguously. Also, for some $S$-values
in which the peak-searching algorithm could not find a peak around the
expected missing-mass of neutron and therefore, we are not presenting any data
points. For the case of $\theta_{2}=134^{\circ}$, the data points of the lower
values of $S$ ($S<40$ MeV) are not presented because the $S$-curve overlapped
with the energy threshold of the ball detectors and we could not separate the
signal from the noise. Note that our measured values for $A_{x}$ for coplanar
configurations ($\phi_{12}=180^{\circ}$) are compatible with zero. This is
consistent with expectations based on parity conservation and, therefore,
shows that our analysis method is reliable. The present results correspond to
the relatively large values of relative energy, $E_{rel}>40$ MeV, between two
final-state protons. Aside from the slight disagreements for the smaller
$\theta_{1}$ ($20^{\circ}$ and $25^{\circ}$) and $\phi_{12}$ ($140^{\circ}$),
the results reveal a fairly good agreement between data and the theoretical
predictions for $A_{x}$ and $A_{y}$, confirming the previous report on a part
of the phase space of this experiment in which two protons scatter to the
forward angles with large azimuthal opening angle MARDANPOUR2010 . It means
that the theoretical models provide a good description of analyzing powers for
the cases with a large relative energy while they fail to describe the data
for lower values of relative energy in proton-deuteron break-up reaction
MARDANPOUR2010 .

For some configurations in which $\theta_{2}=134^{\circ}$, the two 3NF models
provide different values of $A_{y}$ at the intermediate values of $S$.
Considering the statistical and systematic uncertainties, our measurement
cannot differentiate between these models.

Figure 7: Experimental vector analyzing power $A_{x}$ for
$\theta_{2}=45^{\circ}$ and for various combinations of
($\theta_{1},\phi_{12}$), as indicated in each panel. Data are compared with
predictions of various theoretical approaches based on pairwise NN
interactions alone NN (dash lines) as well as with 3NF included NN+3NF (solid
lines); see the legend. The error bars represent statistical uncertainties.
The cyan band depicts the systematic uncertainty ($2\sigma$) which stems from
different sources; see the text.

Figure 8: Same as Figure 7 but for $A_{y}$ instead of $A_{x}$.

Figure 9: Same as Figure 7 but for $\theta_{2}=134^{\circ}$.

Figure 10: Same as Figure 9 but for $A_{y}$ instead of $A_{x}$.

##  5 Summary and conclusion

In this article, a study of the vector analyzing powers in the proton-deuteron
break-up reaction with an incident polarized proton-beam energy of $190$ MeV
impinging on a liquid deuterium target is presented. The data were obtained
exploiting the almost $4\pi$ detection system BINA. The focus was on the part
of the phase space at which one of the protons scattered to intermediate and
large polar angles. The backward-ball of the BINA is used for the first time
to extract break-up observables in coincidence with the forward-wall. We
presented the vector analyzing powers, $A_{x}$ and $A_{y}$, for a set of
kinematical configurations in which one of the protons scatters to the angles
of $\theta_{2}=45^{\circ}$ and $\theta_{2}=134^{\circ}$ and the other to
forward angles ($12^{\circ}<\theta_{1}<32^{\circ}$). This part of the break-up
phase space at this beam energy is presented for the first time. At the large
scattering angle of the second proton, $\theta_{2}=134^{\circ}$, the
theoretical predictions are in fair agreement with the data. However, the two
3NF models provide different values of analyzing powers at some kinematics.
With the precision of the present experiment, one can, however, not
distinguish between these models. The 3NF effects are expected to be small at
the configurations with $\theta_{2}=45^{\circ}$ and at large azimuthal opening
angles $\phi_{12}$. For these configurations the theoretical predictions are
also in good agreement with the data. The results presented here mainly
correspond to larger values of relative energy between two final-state
protons, and the agreement between data and calculations are consistent with
earlier measurements for this range of relative energy. On the other hand, a
significant disagreement between data and theoretical predictions was reported
at this energy for the configurations in which the relative energy is smaller
than $10$ MeV MARDANPOUR2010 . The physics behind this discrepancy is not
clear yet. A similar, but less pronounced, discrepancy has been observed in
the proton-deuteron break-up reaction measured at a proton beam energy of
$130$ MeV Hajar2019 . Further investigation of all parts of the break-up phase
space at different beam energies is needed to have a complete picture of
nuclear forces and its dependence on different underlying dynamics.

##  6 Acknowledgment

We would like to express our gratitude to the AGOR accelerator group at KVI
for delivering the high-quality polarized beam. This work was partly supported
by the Polish National Science Center under Grants No. 2012/05/B/ST-2/02556
and No. 2016/22/M/ST2/00173.

## References

  * (1) J. Chadwick,  Proc. R. Soc. London, Ser. A 136, 692 (1932).
  * (2) H. Yukawa,  Proc. Phys. Math. Soc. Jap. 17, 48 (1935).
  * (3) R. B. Wiringa, V. G. J. Stoks, and R. Schiavilla,  Phys. Rev. C 51, 38 (1995).
  * (4) R. Machleidt,  Phys. Rev. C - Nucl. Phys. 63, 32 (2001).
  * (5) J. Carlson et al.,  Rev. Mod. Phys. 87, 1067 (2015).
  * (6) A. Nogga et al.,  Phys. Rev. C 67, 034004 (2003).
  * (7) A. Nogga, H. Kamada, and W. Glöckle,  Phys. Rev. Lett. 85, 944 (2000).
  * (8) H. Sakai et al.,  Phys. Rev. Lett. 84, 5288 (2000).
  * (9) K. Ermisch et al.,  Phys. Rev. Lett. 86, 5862 (2001).
  * (10) R. V. Cadman et al.,  Phys. Rev. Lett. 86, 967 (2001).
  * (11) K. Hatanaka et al.,  Phys. Rev. C 66, 044002 (2002).
  * (12) J. Carlson, V. Pandharipande, and R. Wiringa,  Nucl. Phys. A 401, 59 (1983).
  * (13) B. S. Pudliner, V. R. Pandharipande, J. Carlson, S. C. Pieper, and R. B. Wiringa,  Phys. Rev. C 56, 1720 (1997).
  * (14) S. Nemoto et al.,  Few-Body Syst. 24, 213 (1998).
  * (15) A. Deltuva, K. Chmielewski, and P. U. Sauer,  Phys. Rev. C 67, 034001 (2003).
  * (16) K. Chmielewski, A. Deltuva, A. C. Fonseca, S. Nemoto, and P. U. Sauer,  Phys. Rev. C 67, 014002 (2003).
  * (17) J. Fujita and H. Miyazawa,  Prog. Theor. Phys. 17, 360 (1957).
  * (18) G. Brown and A. Green,  Nucl. Phys. A 137, 1 (1969).
  * (19) N. Kalantar-Nayestanaki, E. Epelbaum, J. G. Messchendorp, and A. Nogga,  Rep. Prog. Phys. 75, 016301 (2011).
  * (20) S. Kistryn and E. Stephan,  J. Phys. G 40, 063101 (2013).
  * (21) S. Weinberg,  Phys. Lett. B 251, 288 (1990).
  * (22) U. van Kolck,  Phys. Rev. C 49, 2932 (1994).
  * (23) J. L. Friar, D. Hüber, and U. van Kolck,  Phys. Rev. C 59, 53 (1999).
  * (24) R. Machleidt and D. Entem,  Phys. Rep. 503, 1 (2011).
  * (25) LENPIC Collaboration, E. Epelbaum et al.,  Phys. Rev. C 99, 024313 (2019).
  * (26) LENPIC Collaboration, S. Binder et al.,  Phys. Rev. C 98, 014002 (2018).
  * (27) K. Ermisch et al.,  Phys. Rev. C 71, 064004 (2005).
  * (28) D. Hüber and J. L. Friar,  Phys. Rev. C 58, 674 (1998).
  * (29) R. Bieber et al.,  Phys. Rev. Lett. 84, 606 (2000).
  * (30) S. Kistryn et al.,  Phys. Rev. C 72, 044006 (2005).
  * (31) S. Kistryn et al.,  Phys. Lett. B 641, 23 (2006).
  * (32) E. Stephan et al.,  Phys. Rev. C 76, 057001 (2007).
  * (33) H. R. Amir-Ahmadi et al.,  Phys. Rev. C 75, 041001 (2007).
  * (34) A. A. Mehmandoost-Khajeh-Dad et al.,  Eur. Phys. J. A 47, 59 (2011).
  * (35) H. Mardanpour-Mollalar,  Investigation of nuclear forces in d + p elastic and p + d break-up reactions at intermediate energies,  PhD thesis, University of Groningen, 2008.
  * (36) M. Eslami-Kalantari,  Three-nucleon force effects in proton-deuteron break-up studied with BINA at 135 MeV,  PhD thesis, University of Groningen, 2009.
  * (37) A. Ramazani-Moghaddam-Arani,  Cross-section and analyzing-power measurements in three and four-nucleon scattering,  PhD thesis, University of Groningen, 2009.
  * (38) W. Parol et al.,  EPJ Web Conf. 130, 07019 (2016).
  * (39) E. Stephan et al.,  Eur. Phys. J. A 49, 36 (2013).
  * (40) H. Mardanpour et al.,  Phys. Lett. B 687, 149 (2010).
  * (41) G. Ohlsen, R. E. Brown, F. Correll, and R. Hardekopf,  Nucl. Instrum. Methods 179, 283 (1981).
  * (42) M. Morháč, J. Kliman, V. Matoušek, M. Veselský, and I. Turzo,  Nucl. Instrum. Methods A 401, 113 (1997).
  * (43) M. Morháč, J. Kliman, V. Matoušek, M. Veselský, and I. Turzo,  Nucl. Instrum. Methods A 443, 108 (2000).
  * (44) A. Deltuva, R. Machleidt, and P. U. Sauer,  Phys. Rev. C 68, 024005 (2003).
  * (45) W. Glöckle, H. Witała, D. Hüber, H. Kamada, and J. Golak,  Phys. Rep. 274, 107 (1996).
  * (46) J. Kuroś-Żołnierczuk et al.,  Phys. Rev. C 66, 024003 (2002).
  * (47) J. Kuroś-Żołnierczuk et al.,  Phys. Rev. C 66, 024004 (2002).
  * (48) H. Tavakoli-Zaniani et al.,  pre-print (2019), arXiv:1908.10071[nucl-ex].