Audio classification of the content of food containers and drinking glasses
This work is supported by the CHIST-ERA programme through the project CORSMAL, under UK EPSRC grant EP/S031715/1.

Santiago Donaher
Centre for Intelligent Sensing

Queen Mary University of London

London, UK

Alessio Xompero
Centre for Intelligent Sensing

Queen Mary University of London

London, UK

Andrea Cavallaro
Centre for Intelligent Sensing

Queen Mary University of London

London, UK

Food containers, drinking glasses and cups handled by a person generate sounds that vary with the type and amount of their content. In this paper, we propose a new model for sound-based classification of the type and amount of content in a container. The proposed model is based on the decomposition of the problem into two steps, namely  action recognition and content classification. We use the scenario of the recent CORSMAL Containers Manipulation dataset and consider two actions (shaking and pouring), and seven combinations of material and filling level. The first step identifies the action performed by a person with the container. The second step determines the amount and type of content using an action-specific classifier. Experiments show that the proposed  model achieves 76.02, 78.24, and 41.89 weighted average $F_1$ score on the three test sets, respectively, and outperforms baselines and existing approaches that classify the content amount and type either independently or jointly.

Audio classification, Deep Neural Networks, Object properties.

§ INTRODUCTION

The contactless estimation of the weight of a food container or a drinking glass handled by a person is important to support human-robot collaborations when prior knowledge of its content is unavailable <cit.>. The object weight (and hence the force to be applied by the robot hand or gripper for a human-to-robot handover) varies with the amount and type of content. We are interested in inferring this information from the sound generated by the contact of the  content with the surfaces of the container during  a pouring or a shaking action <cit.>.

Previous work on audio regression of content measured the air volume variations within a container <cit.>, the vibrations of liquid poured from a bottle <cit.> and the vibrations in the acoustic frequency generated by a robot shaking and pouring granular materials from and into a tub <cit.>.
Clarke  <cit.> evaluated multiple models, such as a linear regressor, a convolutional neural network, and recurrent neural networks based on either a Long Short-Term Memory (LSTM) module or a Gated Recurrent Unit (GRU) module, to regress the amount of content shaken or poured, using spectrograms as input.
Liang  <cit.> regressed the level of the remaining air column in a container (e.g. glass, cup, thermo, mug) during the pouring of a liquid (water, juice) with a LSTM-based recurrent neural network. To handle noisy scenarios, Liang  <cit.> regressed the liquid level by conditioning on both audio and force/torque data of three containers, and the trained model generalises to different containers.
However, these works specifically addressed the pouring action of containers standing on surfaces.
We are instead interested in the problem of classifying the content of a range of food containers and drinking glasses handled by a person (see Fig. <ref>).

Sample spectrograms (bottom) of the sound produced by the action of pouring  content into a drinking glass (left, centre) and shaking a food container (right).

In this paper, we propose an audio classification model to recognise the properties of the content of containers manipulated by a person. We address the problem by first recognising the action occurring and then identifying an action-specific classifier to determine a set of seven combinations of content type and level.
We validate the proposed model on the CORSMAL Containers Manipulation (CCM) dataset <cit.>, the only publicly available dataset for the recognition of the properties of an object manipulated by a person.  In addition to CCM, we evaluate the proposed model on another set of audio recordings acquired in a different environment and with a different setup[Code, pre-trained models and data are available at: <http://corsmal.eecs.qmul.ac.uk/audio_classification.html>.].

The mass of objects (container and filling) in the training split of the CCM dataset, and the capacity (cap.) of each container. The class empty corresponds to the mass of the container, which is known. key – CX: content type (C) and level (X), where C can be pasta (P), rice (R), or water (W); and X can be Half-full (5) or Full (9).

Empty        2       3       10      15      86      134     15      31       59

P5           37      39      119     37      131     205     104     297      776

P9           65      69      205     54      175     262     174     509      1350

R5           78      85      232     68      208     280     208     633      1548

W5           95      104     270     79      234     316

R9           139     151     409     110     305     396     362     1115     2739

W9           169     185     478     130     352     461

2lCap. (mL)        185     202     520     128     296     363     472     1240     3209

§ PROPOSED METHOD

Let us consider a person that interacts with a container by, for example, shaking a food box or pouring content into a cup or drinking glass. During the interaction, the contact of the content with the container and other content generates different acoustic phenomena. The CORSMAL Containers Manipulation <cit.> dataset covers this scenario with 1,140 audio recordings of 12 people manipulating 15 containers (5 cups, 5 drinking glasses, 5 food boxes) filled with pasta, rice, or water (note that water is not used with food boxes).

While each property (content type, content level) could be classified independently <cit.>, the combination of the two predictions can result in a wrong classification, if either is incorrect. We thus  define a set of seven classes that combine content types and levels, ,  (see Tab. <ref>).
We tackle the task of predicting class $\tilde{c} \in \mathcal{C}$  with a model consisting of three deep neural network classifiers (see Fig. <ref>). We named the proposed model Audio Classification of Content, or ACC.
As pre-processing, we down-sample the signal to 22,050 Hz and, to eliminate relative gain variations, we normalise it to [-1,1]. We perform onset <cit.> detection to identify the beginning of an action, and transform a temporal window of duration $L$ from the onset to a spectrogram $\mtrx{S}$.

The first classifier, $f(\cdot)$, is an action model that estimates the type of container manipulation. We also consider an unknown class that covers cases with  no content. Thus we define the set $\mathcal{A} =\{ pouring, shaking, unknown \}$ with the classes for the action model, $f(\cdot)$, which is trained to classify the action $\boldsymbol{\pi}  \in \{0,1\}^{|\mathcal{A}|}$, where $|\cdot|$ is the cardinality of a set. The action model is a deep neural network that consists of 4 convolutional, 2 max-pooling, and 3 fully connected layers.

The second classifier is selected based on the action: $g(\cdot)$ for pouring and $h(\cdot)$ for shaking. Each specialised classifier estimates the content type and level, and is a deep neural network consisting of 6 convolutional, 3 max-pooling, and 3 fully connected layers for the pouring model; and 4 convolutional, 2 max-pooling, and 2 fully connected layers for the shaking model. For the three models, we use ReLU as activation function after each convolutional and fully connected layer, except for the last layer (softmax); max-pooling layers after every other convolutional layer; and set the kernel size for convolutional and pooling layers to 3$\times$3. The pouring model implements dropout <cit.> in each convolutional layer before the pooling layer to prevent overfitting.

Each specialised classifier outputs a one-hot vector $\mtrx{z} \in \{0,1\}^{|\mathcal{C}|}$, i.e.  and . For pouring, the classifier $g(\cdot)$ predicts the classes $\mathcal{C} \setminus \{$empty$\}$, whereas the classifier for shaking, $h(\cdot)$ predicts the classes $\mathcal{C} \setminus \{$empty, water-half-full, water-full$\}$. As the two specialised classifiers cover  situations with content only, we define an additional 7-dimensional vector $\mtrx{z}_a \in \{0,1\}^{|\mathcal{C}|}$ specifically for the unknown cases when classifying the content properties, and that has only the first element set to 1, i.e. empty container for the CCM dataset.
As the decision of the action classifier conditions the choice of the classifier for the recognition of the content property (or class unknown), we compute the final estimation as

\begin{equation}
\tilde{c} = \left[ \pi_1\mtrx{z}_a + \pi_2\mtrx{z}_g + \pi_3 \mtrx{z}_h \right].
\end{equation}

The proposed model for the  classification of the content type and level, $\tilde{c}$, of a container  manipulated by a person. The model first understands the action $\boldsymbol{\pi}$  based on an  $N \times N$ spectrogram $\mtrx{S}$ and then selects the appropriate branch for pouring or shaking. The size of the feature maps after each pooling layer are shown in brackets at the bottom of the diagram. Legend: Conv: convolutional layer, 2pt[gray,line width=2] (0,0) – (0.3,0); max-pooling layer.

§ VALIDATION

§.§ Methods under comparison

We compare ACC with 9 alternative approaches, namely ResNet (18 layers) <cit.>, a simpler ResNet variant (ResNet$^*$), a ResNet (18 layers) pre-trained on ImageNet (ResNet-18$^\top$) <cit.> and fine-tuned on the training split of CCM, VGG (11 layers) <cit.>, Support Vector Machine (SVM) <cit.>, Random Forest <cit.>, K-Nearest Neighbours (kNN) <cit.>, and the top-2 submissions of the 2020 CORSMAL Challenge[<https://corsmal.eecs.qmul.ac.uk/challenge.html>], namely Because It's Tactile (BIT) <cit.>, and HVRL <cit.>.
SVM, kNN, Random Forest, VGG, and ResNet-based classifiers perform direct classification as a single model.
ResNet$^*$ is a lightweight version of ResNet, consisting of 15 convolutional layers grouped into 3 blocks, with 16, 32, and 64 filters, respectively, and a fully connected layer for the 7 classes; and we also removed the first pooling layer.

BIT <cit.> classifies content level and type independently with a late fusion of audio and RGB data. Audio features provided as input to a Random Forest classifier are Mel-frequency cepstrum coefficients, spectral characteristics, zero crossing rate, chroma vector and deviation; and audio features generated by a modified VGG architecture are provided as input to a GRU-based recurrent neural network. For content-type classification, BIT averages the estimation of the two classifiers. For content-level classification, BIT retrains the two classifiers and includes an additional classifier that uses spatio-temporal features extracted from the RGB frames of the four camera views in CCM. BIT then fuses the decision of the three classifiers.

HVRL <cit.> uses a convolutional neural network based on VGG <cit.> to classify the content type from multiple sequential audio frames converted to spectrogram, followed by majority voting. For each audio frame, the feature map prior to the fully connected layers is processed by an LSTM-based recurrent neural network, followed by a fully connected layer, to temporally estimate the content level. The decision of the last frame is the final classification result.

§.§ Datasets and performance measure

The 1,140 audio recordings of the CCM dataset <cit.> are split into three sets that are evenly distributed among the container types: training (684), public testing (228) and private testing (228). We randomly split the recordings of the training set into training and validation with a 80/20 ratio.

The audio of the CCM dataset was recorded with a circular array of 8 Boya BY-M1 omnidirectional Lavelier microphones and sampling rate 44.1 kHz. The array was placed on the table and the manipulation occurred at a distance between 0.5 to 3 m from the microphones. The multi-channel sound signal is converted into mono channel by averaging samples across channels <cit.>.

We also consider 21 additional recordings acquired in a different room with a Blue Yeti Studio microphone and a sampling rate of 16 bit/48 kHz. We use 4 different containers (two drinking glasses, one cup, one box) and perform a total of 19 pouring and 2 shaking actions with a different type of pasta and rice but the same filling levels as CCM. All recordings were acquired with the microphone placed on a table at a 12 cm from the container for the pouring action and about 20 cm for the shaking action.

To handle class-unbalanced data, we use the weighted average F$_1$ score, $\Bar{F}_1$, as performance measure across the $|\mathcal{C}|$ classes, each weighted by the number of audio recordings, $R_n$, in each class $n$:

\begin{equation}
\Bar{F}_1 = \frac{1}{R} \sum_{n=1}^{|\mathcal{C}|} R_n F_n,
\label{eq:wafs}
\end{equation}

where $R=\sum_{n=1}^{|\mathcal{C}|} R_n$ is the total number of  recordings and $F_n$ is the F$_1$ score of class $n$, i.e. the harmonic mean of precision (the number of true positives over the total number of true positives and false positives) and recall (the number of true positives over the total number of true positives and false negatives) for that class.
We report $\Bar{F}_1$ as a percentage for the discussion of the results.

§.§ Implementation details

We train each classifier independently for 100 epochs with categorical cross-entropy loss and  select the model with the minimum validation loss. During training, we use the Adam <cit.> optimiser with a learning rate of 0.001 and a batch size of 16. Since the classifiers need spectrograms of $N\times N$ resolution as input, we resize the spectrograms to $N=96$. For the temporal segmentation, we also set $L=10$ s based on the average duration of the actions. Note that we complement sound signals with duration shorter than $L$ with zero padding, as other strategies, such as repeating or interpolation, affect the prediction of the content level and worsen the accuracy in validation.

All the deep neural networks that perform direct classification are re-implemented or provided in TensorFlow. These models receive the same input as ACC. SVM, kNN, and Random Forest are provided in scikit-learn and receive a spectrogram reshaped into a 1D vector.

§.§ Discussion

Comparison of the weighted average $F_1$ score ($\Bar{F}_1$) of the combined content type and level classes on the test splits of ccm <cit.> and complexity of the models. Note that bit and hvrl estimate content type and level as two independent properties. best results are in bold, second best in italic.key – # params.: number of parameters.

Model     # Params.     Storage     2cCCM test sets

$\times1000$     MB     Public     Private

Random                 –     –     9.57     11.22

<cit.>           BIT       	           –     –     75.00     77.86

<cit.>         HVRL           	       6,839     82.1     82.14     73.40

<cit.>       kNN                   –     63.5     70.33     62.99

<cit.>       SVM                   –     34.8     69.81     73.43

<cit.>       Random Forest         –     1.7     73.59     70.48

<cit.>                      VGG-11                44,907     175.4     74.03     72.67

<cit.>        ResNet$^*$     179     0.8     70.45     71.56

<cit.>        ResNet-18              11,692     45.8     62.28     55.28

<cit.>        ResNet-18$^\top$       11,692     45.8     59.69     63.01

ACC         16,482     64.3     76.02     78.24

Tab. <ref> compares the weighted average $F_1$ scores of the models under analysis on the CCM test splits, and their complexity in terms of number of parameters and storage size. ACC achieves $\Bar{F}_1$ of 76.02 and 78.24 for the two test splits, respectively. ACC outperforms all the models that directly classify on both test splits.
HVRL is the best performing on the public test split, but its accuracy decrease on the private test split shows that independently estimating the two content properties does not always guarantee consistency of the two predictions. BIT classifies content types and levels independently, and shows a similar behaviour of HVRL but achieving higher weighted average $F_1$ score in the private test split and a lower one in the public test split.

The architecture of ACC is based on that of VGG but with almost 3 times fewer parameters (16,482,637 versus 44,907,000, respectively), even though three independent classifiers are used: the action classifier has 7,077,315 parameters, the classifier specific for pouring has 2,590,150 parameters, and the classifier specific for shaking has 6,815,172 parameters. Moreover, ACC has 5,000,000 more parameters than the two ResNet-18 models, whereas HVRL has a convolutional neural network of  4,472,580 and an LSTM of 2,366,211 parameters, respectively, that requires more storage (82.1 MB). However, the modified ResNet has only 179,000 parameters, less than 1 MB in storage, and achieves higher performance than ResNet-18. Note also how the use of the resized spectrogram as input during training makes the storage of the trained kNN and SVM classifiers comparable to the deep neural network classifiers, while Random Forest better copes with the large input and needs only 1.7 MB to store the trained model.

axis on top,
axis x line*=bottom,
ylabel=True label,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
xlabel=Predicted label,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
xlabel=Predicted label,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
ylabel=True label,
xlabel=Random Forest,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
ylabel=True label,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
ylabel=True label,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=7, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

hide axis,
scale only axis,
height=.05, width=0.3,
anchor=south west,
point meta min=0,
point meta max=1,
colorbar horizontal,
colorbar style=
anchor=south west,
xticklabel pos=lower,
xticklabel style=
/pgf/number format/.cd,
fixed zerofill,
tick label style=font=,
[draw=none] coordinates (0,0);

Confusion matrices for all methods across all the containers of the public and private testing splits of CCM <cit.>. KEY – 0: empty; CX: content type (C) and level (X), where C represents pasta (P), rice (R), or water (W); and X is  half-full (5) or full (9).

Fig. <ref> compares the confusion matrices for the models performing classification of the combined content type and level on the combined CCM test splits. ResNet-18 and ResNet-18$^\top$ wrongly classifies pasta-half-full with pasta-full, and transfer learning helps to reduce the wrong classification of water-half-full with water-full. Reducing the number of layers from 18 to 15 helps to increase the correct classification across the different classes, while decreasing the overfitting on this small-size dataset. Overall, the methods tend to wrongly classify the level half-full with full and the opposite, while the empty (unknown) case is correctly classified by all methods. ACC reduces the wrong classifications and achieves a higher number of correct classifications along the diagonal compared to the other models.

Comparison of the Weighted Average $F_1$ score ($\Bar{F}_1$) on the testing set with 21 audio recordings of novel containers used in a different environment and setup. Best results are in bold, second best in italic.

2cModel     Level    Type     Type & Level

Random       	    43.98	    17.05	      15.24

<cit.>	     kNN          	    37.28	    37.06	     19.09

<cit.>	     SVM          	    40.77	     47.39    	     26.98

<cit.>	     Random Forest        47.62	    44.49	     20.07

<cit.>	     VGG-11             	    58.53	    25.95	     15.24

<cit.> 	     ResNet$^*$          	    52.04    	56.11	    34.95

<cit.> 	     ResNet-18          	    50.40	    45.31	     30.87

<cit.> 	     ResNet-18$^\top$      	    50.30	    61.50	     32.02

ACC          	     58.34     	    53.85	     41.89

Tab. <ref> compares the results obtained from the recordings of the second, additional setup. ACC achieves $\Bar{F}_1=58.34$ in correctly predicting only the content level, outperforming the other approaches except VGG-11 ($\Bar{F}_1=58.53$). For content type, ACC achieves $\Bar{F}_1=53.85$ and only ResNet-18 and ResNet$^*$ achieves higher accuracy ($\Bar{F}_1=61.50$ and $56.11$, respectively). Despite this difference between the accuracy for content type and content level, ACC outperforms other approaches when validating the combination of both content type and level. ACC achieves $\Bar{F}_1=41.89$, while the second best, ResNet$^*$, achieves $\Bar{F}_1=34.95$.

Fig. <ref> shows the confusion matrices of the action classifier for the combined test sets in CCM and the test set acquired in a different environment and setup. The action classifier of ACC can correctly distinguish between the two actions of pouring and shaking, as well as the unknown case, for unknown containers in the same setup and environment (CCM). The prediction is instead more challenging in the novel setup and environment, as unknown is confused with pouring and some recordings with pouring are confused with shaking.
The higher sensitivity and closer position of the Blue Yeti Studio microphone compared to the Boya BY-M1 omnidirectional Lavelier microphone are possible causes of the lower performance in addition to the type of content.

axis on top,
axis x line*=bottom,
ylabel=True label,
y dir=reverse,
point meta min=0,
point meta max=1,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=3, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
xlabel=Predicted label,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

axis on top,
axis x line*=bottom,
ylabel=True label,
y dir=reverse,
tick label style=font=,
label style=font=,
[matrix plot*,mesh/cols=3, point meta=explicit] table [meta=z] ;

axis x line*=top,
axis y line*=right,
y dir=reverse,
xlabel=Predicted label,
axis on top,
point meta min=0,
point meta max=1,
tick label style=font=,
yticklabel style = ,
label style=font=,

hide axis,
scale only axis,
height=.05, width=0.3,
anchor=south west,
point meta min=0,
point meta max=1,
colorbar horizontal,
colorbar style=
anchor=south west,
xticklabel pos=lower,
xticklabel style=
/pgf/number format/.cd,
fixed zerofill,
tick label style=font=,
[draw=none] coordinates (0,0);

Confusion matrices for the action classifier of ACC across all the containers of the testing split in CCM (left) and the novel containers (NC) used in the second setup (right). KEY – U: unknown, P: pouring, S: shaking.

§ CONCLUSION

We presented an audio classification model that recognises the content type and level in a container that  is manipulated by a person who shakes it or  pours content into it. The proposed model works with both liquid and granular materials and classifies the  properties of a range of unknown containers with different materials, shapes and sizes. The proposed model outperforms existing solutions for the estimation of content type and level independently, as well as baselines based on SVM, kNN, random forest and deep neural networks that directly classify content type and level. As future work, we will extend the proposed model to generalise to different setups and environments, and to cope with multi-modal data <cit.>.


# Direct mass measurements of neutron-rich zinc and gallium isotopes: an
investigation of the formation of the first r-process peak

Andrew Jacobs<EMAIL_ADDRESS>TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T
1Z1, BC, Canada  Department of Physics and Astronomy, University of British
Columbia, 6224 Agricultural Rd, Vancouver, V6T 1Z1, BC, Canada  Stylianos
Nikas  University of Jyvaskyla, P.O. Box 35, Fl-40014 University of Jyvaskyla,
Finland  John Ash  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada
Behnam Ashrafkhani  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada
Department of Physics and Astronomy, University of Calgary, 2500 University
Drive NW, Calgary, T2N 1N4, Alberta, Canada  Ivana Belosovic  TRIUMF, 4004
Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Julian Bergmann  II.
Physikalisches Institut, Justus-Liebig-Universität Gießen, Gießen, 35392,
Hessen, Germany  Callum Brown  Institute for Particle and Nuclear Physics,
University of Edinburgh, Peter Guthrie Tait Road, Edinburgh, EH9 3FD, United
Kingdom  Jaime Cardona  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC,
Canada  Department of Physics and Astronomy, University of Manitoba, 30A
Sifton Road, Winnipeg, R3T 2N2, Manitoba, Canada  Eleanor Dunling  TRIUMF,
4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Timo Dickel  II.
Physikalisches Institut, Justus-Liebig-Universität Gießen, Gießen, 35392,
Hessen, Germany  GSI Helmholtzzentrum für Schwerionenforschung GmbH, 64291
Darmstadt, Germany  Gabriella Gelinas  TRIUMF, 4004 Wesbrook Mall, Vancouver,
V6T 1Z1, BC, Canada  Department of Physics and Astronomy, University of
Calgary, 2500 University Drive NW, Calgary, T2N 1N4, Alberta, Canada  Zach
Hockenbery  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada
Department of Physics, McGill University, 3600 Rue University, Montréal, QC
H3A 2T8, Canada  Sakshi Kakkar  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T
1Z1, BC, Canada  Department of Physics and Astronomy, University of Manitoba,
30A Sifton Road, Winnipeg, R3T 2N2, Manitoba, Canada  Brian Kootte  TRIUMF,
4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Department of Physics and
Astronomy, University of Manitoba, 30A Sifton Road, Winnipeg, R3T 2N2,
Manitoba, Canada  Ali Mollaebrahimi  TRIUMF, 4004 Wesbrook Mall, Vancouver,
V6T 1Z1, BC, Canada  II. Physikalisches Institut, Justus-Liebig-Universität
Gießen, Gießen, 35392, Hessen, Germany  Eleni Marina Lykiardopoulou  TRIUMF,
4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Department of Physics and
Astronomy, University of British Columbia, 6224 Agricultural Rd, Vancouver,
V6T 1Z1, BC, Canada  Tobias Murböck  TRIUMF, 4004 Wesbrook Mall, Vancouver,
V6T 1Z1, BC, Canada  II. Physikalisches Institut, Justus-Liebig-Universität
Gießen, Gießen, 35392, Hessen, Germany  Stefan Paul  TRIUMF, 4004 Wesbrook
Mall, Vancouver, V6T 1Z1, BC, Canada  Wolfgang R. Plaß  II. Physikalisches
Institut, Justus-Liebig-Universität Gießen, Gießen, 35392, Hessen, Germany
GSI Helmholtzzentrum für Schwerionenforschung GmbH, 64291 Darmstadt, Germany
William S. Porter  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada
Department of Physics and Astronomy, University of Notre Dame, Notre Dame, IN
46556, USA  Moritz Pascal Reiter  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T
1Z1, BC, Canada  Institute for Particle and Nuclear Physics, University of
Edinburgh, Peter Guthrie Tait Road, Edinburgh, EH9 3FD, United Kingdom  Alex
Ridley  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Jon
Ringuette  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada
Department of Physics, Colorado School of Mines, Golden, Colorado 80401, USA
Christoph Scheidenberger  II. Physikalisches Institut, Justus-Liebig-
Universität Gießen, Gießen, 35392, Hessen, Germany  GSI Helmholtzzentrum für
Schwerionenforschung GmbH, 64291 Darmstadt, Germany  Helmholtz Research
Academy Hesse for FAIR (HFHF), GSI Helmholtz Center for Heavy Ion Research,
Campus Gießen, 35392 Gießen, Germany  Rane Simpson  TRIUMF, 4004 Wesbrook
Mall, Vancouver, V6T 1Z1, BC, Canada  Department of Physics and Astronomy,
University of British Columbia, 6224 Agricultural Rd, Vancouver, V6T 1Z1, BC,
Canada  Coulter Walls  TRIUMF, 4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC,
Canada  Department of Physics and Astronomy, University of Manitoba, 30A
Sifton Road, Winnipeg, R3T 2N2, Manitoba, Canada  Yilin Wang  TRIUMF, 4004
Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Department of Physics and
Astronomy, University of British Columbia, 6224 Agricultural Rd, Vancouver,
V6T 1Z1, BC, Canada  Jens Dilling  Oak Ridge Natonal Laboratory, 1 Bethel
Valley Road, Oak Ridge, 37831, TN, United States  Ania Kwiatkowski  TRIUMF,
4004 Wesbrook Mall, Vancouver, V6T 1Z1, BC, Canada  Department of Physics and
Astronomy, University of Victoria, 3800 Finnerty Road, Victoria, V8P 5C2, BC,
Canada

###### Abstract

The prediction of isotopic abundances resulting from the rapid neutron capture
process (r-process) requires high-precision mass measurements. Mass
measurements of 79-83Zn and 85,86Ga using TITAN’s on-line time-of-flight
spectrometer. First time measurements are performed for 79mZn, 83Zn, and 86Ga.
These measurements reduced uncertainties, and are used to calculate isotopic
abundances near the first r-process abundance peak using astrophysical
conditions present during a binary neutron star (BNS) merger. Good agreement
across a range of trajectories is found when comparing to several metal-poor
stellar abundances. Particularly, this subset of trajectories produces
agreement with the abundance pattern of both the ‘r/s-star’ HD94028 as well as
the ‘r-process star’ HD222925. These findings point to a high degree of
sensitivity to the electron fraction of a BNS merger on the final elemental
abundance pattern near the first r-process peak. In particular, we find that
small changes in electron fraction produce distinct abundance patterns that
match those of metal-poor stars with different classifications, calls the need
for an i-process into question.

The formulation of the synthesis of elements heavier than Fe identifies two
processes corresponding to neutron captures [1]. They differentiate between an
environment with a low neutron density ($\approx 10^{6}-10^{10}$ cm-3)
resulting in a slow capture process (s-process) [2], and an environment with a
high neutron density ($\approx 10^{20}$ cm-3) leading to a rapid capture
process (r-process) [3]. Since then, both the s- and r-processes have been
separated into ‘main’ and ‘weak’ components with different astrophysical sites
correlating to each [4, 5, 6, 7, 8, 9].

Significant developments have been achieved regarding the r-process following
the detection of the Binary Neutron Star (BNS) merger GW170917 [10] and its
subsequent kilonova AT2017gfo [11, 12]. From the electromagnetic spectrum, an
early blue emission indicates a moderate r-process environment where isotope
production up to $A\leq 140$ occurred. Furthermore, Sr isotopes specifically
have been identified in the spectra [13] proving the production of light
elements around the 1${}^{\textrm{st}}$ abundance peak. The blue emission is
followed by a red emission which indicates the production of heavy elements up
to lanthanides. Due to the path the r-process takes, nucleosynthesis modeling
often requires the use of nuclear models to provide the necessary nuclear
inputs where data is lacking. As such, it is imperative to provide
experimental data for key nuclei along the r-process path, particularly mass
values [14, 15, 16, 17, 18]. Conversely, occurring close to the ‘valley of
beta stability’ where almost all needed nuclear data is available, the
s-process is comparatively well understood [19].

However, both s- and r-process models fail to describe the isotopic abundances
of pre-solar grains [20, 21, 22, 23] and elemental abundance patterns found in
particular stars (termed ‘r/s-stars’ or ‘i-stars’) [24, 25, 26]. Two notable
inconsistencies found between solar r-process abundance predictions and
r/s-stars are the Ge-As-Se triplet [27, 28] and the enhanced abundances of
elements from $38\leq Z\leq 48$. A solution was developed by utilizing a
neutron capture process, termed the intermediate neutron capture process
(i-process) [29, 30, 31, 32]. Despite the i-process successfully reproducing
stellar abundances between the 1${}^{\textrm{st}}$ and 2${}^{\textrm{nd}}$
r-process abundance peaks, where combinations of s- and r-processes only have
previously failed [33], and demonstrating the production of actinides in low
metallicity AGB stars [34], its existence and validity remains under
discussion [35, 36, 37].

In this Letter we report the first direct mass measurements of 83Zn and 86Ga
as well as an improved mass uncertainty of 85Ga. We revisit the formation of
elements from $32\leq Z\leq 48$ with state-of-the-art r-process
nucleosynthesis calculations in an astrophysical environment conducive to that
of a BNS merger. We show that, when including new experimental values, the
blue kilonova can reproduce the previously challenging abundance ratios of
both: r-process enhanced stars such as HD222925 [38] and typical r/s-stars
which motivate the existence of the i-process such as HD94028 [33].

High precision mass measurements were performed at TRIUMF’s Ion Trap for
Atomic and Nuclear science (TITAN) [39] located in the Isotope Separator and
ACcelerator (ISAC) [40] facility at TRIUMF in Vancouver, Canada. This facility
provides access to short-lived radioactive isotopes involved in these
processes. In this Letter, we report the first science results from the novel
target developed at TRIUMF-ISAC utilizing a W-neutron converter [41].

A resonant laser ionization [42] was used for Zn, and standard surface
ionization was employed for other species [43]. The radioactive ion beam (RIB)
was reduced to a single A/q by a magnetic dipole mass separator [44] and
delivered to TITAN in order to perform a high precision mass measurement using
the MR-TOF-MS.

The TITAN MR-TOF-MS [45, 46] is based on the Gießen-GSI design used at the FRS
Ion-Catcher [47, 48, 49]. The measurements were performed following an
established procedure [50, 46]. In addition, a newly developed technique was
used to achieve the needed beam purity. To perform this operation, the
potentials of the MR-TOF-MS mirror closer to the injection trap can be
lowered, and the ions are dynamically re-captured in the injection trap. This
process, called mass selective re-trapping [51, 52], enables the suppression
of isobaric contamination by up to four orders of magnitude and has been
utilized in several on-line experiments at TITAN [53, 54, 55, 56].

Figure 1: The spectrum and fit (red) for A/q = 83 at 900 turns after 2.5 hours accumulation. 83Ga+ has been used as the calibration species. Species | Calibrant | Mass Ratio | ME${}_{\textrm{TITAN}}$ (keV/c2) | ME${}_{\textrm{AME2020}}$ (keV/c2) | Difference (keV/c2)
---|---|---|---|---|---
79Zn |  79Ga | 1.000136781 (51) | -53430.0 (79) | -53432.3 (22) | 2.3 (8.4)
79mZn |  79Ga | 1.00013678 (16) | -52491 (14) | -52330 (150) | 161 (151)
80Zn |  80Rb | 1.000275561 (48) | -51661 (10) | -51648.6 (26) | -12 (11)
81Zn |  81Ga | 1.00015146 (11) | -46209 (12) | -46200 (5) | -9 (13)
82Zn |  82Ga | 1.000139116 (91) | -42312 (10) | -42314 (3) | 2 (10)
83Zn |  83Ga | 1.00016756 (22) | -36311 (18) | ### | ###
85Ga |  85Rb | 1.00053666 (15) | -39720 (14) | -39740 (40) | 20 (42)
86Ga |  86Rb | 1.00061203 (23) | -33768 (20) | ### | ###
Table 1: Ga and Zn mass measurements with their respective calibrants, mass
ratios ($m_{species}/m_{cal}$), and mass excess results
(ME${}_{\textrm{TITAN}}$) compared to the most recent AME2020 mass excess
(ME${}_{\textrm{AME2020}}$). All species were measured in the singly charged
(1+) state. Mass Excesses (ME) and mass ratios are reported as atomic masses.
Previously unmeasured species are marked as ### .

To extract the masses of the species of interest, a data analysis procedure
using the methodologies described in [57, 58] was performed. A resulting
spectrum in which the mass of 83Zn was determined can be seen in figure 1 To
maximize the resolving power of the system, a time-resolved calibration (TRC)
is done [57]. For the first time, a newly developed ion manipulation technique
referred to as ‘post re-trapping beam merging’ was used to ensure that a TRC
species was available, regardless of RIB composition. This technique is
performed between the isobar separation and mass measurement steps. This
allows the merging of the isobarically purified ion ensemble with a controlled
amount of an additional stable calibration species. In this particular
experiment, post re-trapping beam merging was used to achieve a sufficient
resolving power needed to separate 86Rb+ and 86mRb+ via a 85Rb+ TRC. As such,
an isobaric calibration species is provided for 86Ga+. This, in turn, made the
first high-precision measurement of 86Ga+ possible. Details on this novel
technique are described in [52].

The emgfit Python package [59] was then used to fit the peak centroids using a
hyper-exponentially modified Gaussian lineshape [60]. From the centroids, a
well-measured calibration species is used to extract the final mass of the
species of interest. Statistical and systematic uncertainties are determined
using the procedures described in [57, 58]. Ultimately, a systematic
uncertainty of $\delta m/m=1\cdot 10^{-7}$ has been determined for these
measurements resulting in absolute uncertainties of $\delta m<20$ keV/$c^{2}$.

Table 1 provides a list of the measured masses for 79-83Zn and 85,86Ga. The
calibration species as well as the mass ratios are reported to account for any
adjustment in the calibrant masses in the future. The mass excess results
(ME${}_{\textrm{TITAN}}$), previous high precision measurements
(ME${}_{\textrm{AME2020}}$) [61], and the difference between the two are
reported.

All results are in good agreement with literature values. The uncertainty of
85Ga, reported by TITAN in [62], has been reduced by a factor $\approx$ 3 and
is in agreement with the previous value. Additionally, the excitation energy
of 79mZn was measured for the first time using direct mass spectrometry
techniques, and its uncertainty has been reduced by an order of magnitude.
Lastly, first direct measurements of 83Zn and 86Ga have been performed.

To interpret the impact of these results, we use the statistical model code
TALYS [63] to calculate the corresponding (n,$\gamma$) reaction rates with the
new mass values. Two different mass tables were used. The first mass table
used experimental nuclear masses from AME2020 and, where experimental data is
unavailable, the FRDM2012 mass tables [64] (referred to henceforth as
AME+FRDM). The second mass table used the same combination of AME2020 and
FRDM2012, but included the new and improved TITAN mass values in place of
either AME2020 or FRDM2012 values (referred to as TITAN).

The transmission coefficients for the formation of the compound nucleus are
calculated from a spherical neutron-nucleus optical potential using the
Koning-Delaroche globally fitted parameters [65]. The Back-Shifted Fermi Gas
model is used [66, 67] to describe level densities, and the Kopecky-Uhl
generalized Lorentzian [68] is used for the $\gamma$ strength functions. The
reaction rates were calculated for each mass table using the corresponding
central mass values to calculate the reaction Q-values. A set of 100 random
variations of the masses within the $1\sigma$ uncertainty band of known masses
and the reported error of $\sigma_{th}=0.5595$ MeV from FRDM2012 [64] were
generated in order to probe the effect of the isotope’s mass uncertainty for
each astrophysical trajectory.

The nuclear reaction network code GSINet was used to model the r-process and
calculate the elemental abundances [69]. We initialized calculations at
Nuclear Statistical Equilibrium (NSE), setting the initial temperature as
$T_{0}=10$ GK. Due to the uncertainties in the astrophysical conditions that
could produce nuclei in the first r-process peak region ($A\approx 82$), we
simulated nucleosynthesis for a set of parameterized r-process trajectories
where we treat initial electron fraction ($Y_{e}=N_{p}/(N_{p}+N_{n}$)) as a
free parameter to investigate. This range is encompased by expected electron
fractions found in BNS mergers.

The expansion time scale was set as $\tau=7$ ms, and the initial specific
entropy as $15\ k_{B}/$ baryon. We assume that the density evolution of the
ejecta initially follows an exponential expansion and at later times a
homologous expansion [70]. We evolve our abundances assuming NSE for $T\geq 6$
GK, before changing to the full nuclear network calculations.

Previous work at the first r-process peak has shown a remarkable agreement
between parts of the solar r-process abundances and predictions from r-process
network calculations [62]. However, due to its age, the Sun has likely
undergone several enrichment events [71]. As such, metal-poor (i.e. old) stars
offer additional insights due to experiencing fewer enrichment events. In this
work we explore the abundances of the r-process star HD222925 [38], which has
one of the most complete abundance sets for any object beyond the solar
system, and the r/s-star HD94028 [33] which has been a key motivator for the
existence of the i-process.

Following the calculation of each elemental abundance, the results were
normalized for each value of $Y_{e}$ to the abundance of Se ($Z=34$). To allow
comparison, this procedure was also applied to the measured abundance patterns
of HD94028 [33] and HD222925 [38]. In Figure 2, the resulting abundances for a
selection of $Y_{e}$s is shown from $Z=32-48$ compared to the observed
abundance patterns. We find that, by varying $Y_{e}$ in a narrow range, the
resulting abundance pattern can better match that of either HD94028 or
HD222925 ($Y_{e}=0.336$ and $0.354$ respectively, which is within the range of
$Y_{e}$ estimated from a blue kilonova [72, 73]). However, a strong odd-even
effect from $Z=38-40$ is not reproduced for any value of $Y_{e}$, and the
optimal $Y_{e}$ for reproducing the abundance pattern of HD222925 at $Z=32-34$
drastically under-produces elements of $Z>42$. In the lower half of Figure 2,
the residuals between HD94028 and HD222925 and their optimal $Y_{e}$
abundances are shown normalized to the stellar abundance uncertainty
($\sigma$). Very good agreement is found for HD94028 with the exception of Y
($Z=39$), and HD222925 is somewhat well reproduced with systematic
underproduction beyond $Z=42$.

Figure 2: (Top) Calculated elemental abundances of the 1st r-process for a
wide range of $Y_{e}$ normalized to Se using AME2020 and new TITAN mass data
compared to the abundance patterns of HD94028 (r/s-star), and HD222925
(r-star). (Bottom) The difference in abundances for HD94028 and HD222925 from
their optimal $Y_{e}$ normalized to the 1$\sigma$ values for the stellar
abundances. Thus, within the $Y_{e}$ range of a blue kilonva, the abundance
patterns for two different stellar classifications can be reproduced.

In Figure 3, the abundance results are compared between the two different
datasets (AME+FRDM and TITAN). In the upper panel, the differences in
abundance between AME+FRDM and TITAN are shown for the same $Y_{e}$’s as those
seen in Figure 2 (again normalized to Se). In general, the updated TITAN mass
values predict an increase in Ge and As ($Z=32$ and $33$), a decrease in Br
($Z=35$), and an increase in all elements beyond by $\approx 20\%$. In the
lower panel, the change in abundance is shown for $Y_{e}=0.34$ with the
respective uncertainties of both datasets. The new mass values reduce the
abundance uncertainty by a factor of 2. This increased production beyond
$Z=35$ due to new mass values can partially explain the anomalous increase in
abundance for elements between the 1${}^{\textrm{st}}$ and 2${}^{\textrm{nd}}$
r-process peaks for stars such as HD94028.

Figure 3: (Top) The difference in abundance between the TITAN and AME+FRDM
datasets for a range of $Y_{e}$ matching that seen in figure 2. (Bottom) The
percent difference between TITAN and AME+FRDM results for $Y_{e}=0.34$ and
their corresponding relative uncertainties to 1$\sigma$. This systematic
increase in production for $Z\geq 36$ aids in addressing the previously
thought underproduction by the r-process for these elements.

Certain abundance discrepancies found in stars such as HD94028 have resulted
in their classification as r/s-stars [33], namely the high abundances of Ge
and As relative to Se as well as increased abundances of elements from $38\leq
Z\leq 48$. Hence, we investigate the interplay between these key elemental
ratios. Figure 4 compares the logarithmic ratios of Ge/Se, As/Se, and Se/Zr
for our r-process calculations using the new TITAN mass results (for
trajectories of $Y_{e}=0.33-0.36$), HD94028, HD222925, and several other metal
poor stars with reported abundances of Ge, As, Se, and Zr (from [74, 28]). In
both plots, we find that our results, across a narrow range of $Y_{e}$, match
the general distribution of these metal-poor stars. Moreover, we find that the
optimal $Y_{e}$ for the Ge-As-Se triplet matches the optimal $Y_{e}$ of the
Ge-Se-Zr triplet. With such excellent agreement and high sensitivity to
$Y_{e}$, we propose that the variability of $Y_{e}$ found in a BNS merger can
explain seemingly stark differences in stellar abundance patterns around the
1${}^{\textrm{st}}$ abundance peak without requiring an entirely separate
elemental production mechanism.

In summary, we have performed high precision mass measurements of neutron rich
Zn and Ge isotopes including first time measurements of 83Zn and 86Ga. These
measurements were enabled by the development of the new ISAC proton-to-neutron
converter target as well as the MR-TOF-MS beam merging technique. With these
new mass values, state-of-the-art r-process network calculations were
performed for trajectories conducive to the astrophysical environment of a BNS
merger’s neutrino drive winds which produce a weak r-process. We find that, by
varying the initial electron fraction, $Y_{e}$, we can reproduce the abundance
patterns of stars with different classifications (namely r/s-star HD94028 and
r-process star HD222925). Furthermore, we find that by varying $Y_{e}$, we
reproduce the correlations between key elemental ratios (Ge/Se, As/Se, and
Se/Zr) of several metal-poor stars. Ultimately, these results point to the
possibility that, for a given astrophysical site, small differences in $Y_{e}$
can explain large observed deviations in abundance patterns near the first
r-process peak. As such, this result calls into question the explicit need for
an i-process to explain the observed abundances in r/s-stars such as HD94028.
To further test these conclusions, additional abundance observations from
HD94028 and HD222925 for $35\leq Z\leq 37$ and from $32\leq Z\leq 48$ of
metal-poor stars (such as those in Reticulum II [75]) would prove particularly
useful.

Figure 4: Comparison of logarithmic ratios of key element pairs between TITAN
results (with an additional color scale axis for $Y_{e}=0.33-0.36$ in steps of
$0.002$) and observations of various metal-poor stars. The trend in ratios
from observational stellar data matches that of the $Y_{e}$ range produced by
a blue kilonova.

We would like to thank the Targets and Ion Source group at TRIUMF.
Particularly, thank J. Lassen for the Zn laser development as well as A.
Gottberg and L. Egoriti for the proton-to-neutron converter target
development. Additionally, we would like to thank N. Vassh for fruitful
discussion pertaining to our results. This work was supported by the Natural
Sciences and Engineering Research Council (NSERC) of Canada under Grants No.
SAPIN-2018-00027, No. RGPAS-2018-522453, and No. SAPPJ-2018-00028, the
National Research Council (NRC) of Canada through TRIUMF, the Canada-UK
Foundation, German institutions DFG (grants FR 601/3-1, SCHE 1969/2-1 and SFB
1245 and through PRISMA Cluster of Excellence), BMBF (grants 05P19RGFN1 and
05P21RGFN1), and by the JLU and GSI under the JLU-GSI strategic Helmholtz
partnership agreement.

## References

  * Burbidge  _et al._ [1957] E. M. Burbidge, G. R. Burbidge, W. A. Fowler, and F. Hoyle, Synthesis of the elements in stars, Rev. Mod. Phys. 29, 547 (1957).
  * Käppeler  _et al._ [2011] F. Käppeler, R. Gallino, S. Bisterzo, and W. Aoki, The $s$ process: Nuclear physics, stellar models, and observations, Rev. Mod. Phys. 83, 157 (2011).
  * Horowitz  _et al._ [2019] C. J. Horowitz, A. Arcones, B. Côté, I. Dillmann, W. Nazarewicz, I. U. Roederer, H. Schatz, A. Aprahamian, D. Atanasov, A. Bauswein, T. C. Beers, J. Bliss, M. Brodeur, J. A. Clark, A. Frebel, F. Foucart, C. J. Hansen, O. Just, A. Kankainen, G. C. McLaughlin, J. M. Kelly, S. N. Liddick, D. M. Lee, J. Lippuner, D. Martin, J. Mendoza-Temis, B. D. Metzger, M. R. Mumpower, G. Perdikakis, J. Pereira, B. W. O’Shea, R. Reifarth, A. M. Rogers, D. M. Siegel, A. Spyrou, R. Surman, X. Tang, T. Uesaka, and M. Wang, r-process nucleosynthesis: connecting rare-isotope beam facilities with the cosmos, Journal of Physics G: Nuclear and Particle Physics 46, 083001 (2019).
  * Prantzos  _et al._ [1990] N. Prantzos, M. Hashimoto, and K. Nomoto, The s-process in massive stars-yields as a function of stellar mass and metallicity, Astron. Astrophys. 234, 211 (1990).
  * Busso  _et al._ [1999] M. Busso, R. Gallino, and G. Wasserburg, Nucleosynthesis in asymptotic giant branch stars: Relevance for galactic enrichment and solar system formation, Annual Review of Astronomy and Astrophysics 37, 239 (1999).
  * Winteler  _et al._ [2012] C. Winteler, R. Kaeppeli, A. Perego, A. Arcones, N. Vasset, N. Nishimura, M. Liebendoerfer, and F.-K. Thielemann, Magnetorotationally driven supernovae as the origin of early galaxy r-process elements?, Astrophys. J. Lett. 750, L22 (2012).
  * Wanajo [2013] S. Wanajo, The r-process in proto-neutron-star wind revisited, Astrophys. J. Lett. 770, L22 (2013).
  * Wanajo  _et al._ [2018] S. Wanajo, B. Müller, H.-T. Janka, and A. Heger, Nucleosynthesis in the innermost ejecta of neutrino-driven supernova explosions in two dimensions, Astrophys. J. 852, 40 (2018).
  * Siegel  _et al._ [2019] D. M. Siegel, J. Barnes, and B. D. Metzger, Collapsars as a major source of r-process elements, Nature 569, 241 (2019).
  * Abbott  _et al._ [2017a] B. P. Abbott, R. Abbott, T. Abbott, F. Acernese, K. Ackley, C. Adams, T. Adams, P. Addesso, R. Adhikari, V. B. Adya, _et al._ , GW170817: observation of gravitational waves from a binary neutron star inspiral, Phys. Rev. Lett. 119, 161101 (2017a).
  * Abbott  _et al._ [2017b] B. P. Abbott, R. Abbott, T. Abbott, F. Acernese, K. Ackley, C. Adams, T. Adams, P. Addesso, R. Adhikari, V. Adya, _et al._ , Gravitational waves and gamma-rays from a binary neutron star merger: GW170817 and GRB 170817A, Astrophys. J. Lett. 848, L13 (2017b).
  * Kasen  _et al._ [2017] D. Kasen, B. Metzger, J. Barnes, E. Quataert, and E. Ramirez-Ruiz, Origin of the heavy elements in binary neutron-star mergers from a gravitational-wave event, Nature 551, 80 (2017).
  * Watson  _et al._ [2019] D. Watson, C. J. Hansen, J. Selsing, A. Koch, D. B. Malesani, A. C. Andersen, J. P. Fynbo, A. Arcones, A. Bauswein, S. Covino, _et al._ , Identification of strontium in the merger of two neutron stars, Nature 574, 497 (2019).
  * Brett  _et al._ [2012] S. Brett, I. Bentley, N. Paul, R. Surman, and A. Aprahamian, Sensitivity of the r-process to nuclear masses, The European Physical Journal A 48, 1 (2012).
  * Aprahamian  _et al._ [2014] A. Aprahamian, I. Bentley, M. Mumpower, and R. Surman, Sensitivity studies for the main r process: nuclear masses, AIP Advances 4, 041101 (2014), https://doi.org/10.1063/1.4867193 .
  * Mumpower  _et al._ [2015a] M. R. Mumpower, R. Surman, D.-L. Fang, M. Beard, P. Möller, T. Kawano, and A. Aprahamian, Impact of individual nuclear masses on $r$-process abundances, Phys. Rev. C 92, 035807 (2015a).
  * Mumpower  _et al._ [2015b] M. Mumpower, R. Surman, D. L. Fang, M. Beard, and A. Aprahamian, The impact of uncertain nuclear masses near closed shells on the r-process abundance pattern, J. Phys. G Nucl. Part. Phys. 42, 034027 (2015b).
  * Jiang  _et al._ [2021] X. F. Jiang, X. H. Wu, and P. W. Zhao, Sensitivity study of r-process abundances to nuclear masses, Astrophys. J. 915, 29 (2021).
  * Reifarth  _et al._ [2014] R. Reifarth, C. Lederer, and F. Käppeler, Neutron reactions in astrophysics, Journal of Physics G: Nuclear and Particle Physics 41, 053101 (2014).
  * Fujiya  _et al._ [2013] W. Fujiya, P. Hoppe, E. Zinner, M. Pignatari, and F. Herwig, Evidence for radiogenic sulfur-32 in type AB presolar silicon carbide grains?, Astrophys. J. Lett. 776, L29 (2013).
  * Jadhav  _et al._ [2013] M. Jadhav, M. Pignatari, F. Herwig, E. Zinner, R. Gallino, and G. R. Huss, Relics of ancient post-AGB stars in a primitive meteorite, Astrophys. J. Lett. 777, L27 (2013).
  * Liu  _et al._ [2014] N. Liu, M. R. Savina, A. M. Davis, R. Gallino, O. Straniero, F. Gyngard, M. J. Pellin, D. G. Willingham, N. Dauphas, M. Pignatari, _et al._ , Barium isotopic composition of mainstream silicon carbides from murchison: Constraints for s-process nucleosynthesis in asymptotic giant branch stars, Astrophys. J. 786, 66 (2014).
  * Liu  _et al._ [2019] N. Liu, T. Stephan, S. Cristallo, R. Gallino, P. Boehnke, L. R. Nittler, C. M. Alexander, A. M. Davis, R. Trappitsch, M. J. Pellin, _et al._ , Presolar silicon carbide grains of types Y and Z: Their molybdenum isotopic compositions and stellar origins, Astrophys. J. 881, 28 (2019).
  * Jonsell  _et al._ [2006] K. Jonsell, P. Barklem, B. Gustafsson, N. Christlieb, V. Hill, T. C. Beers, and J. Holmberg, The Hamburg/ESO R-process enhanced star survey (HERES)-III. HE 0338-3945 and the formation of the r+s stars, Astron. Astrophys. 451, 651 (2006).
  * Lugaro  _et al._ [2012] M. Lugaro, A. I. Karakas, R. J. Stancliffe, and C. Rijs, The s-process in asymptotic giant branch stars of low metallicity and the composition of carbon-enhanced metal-poor stars, Astrophys. J. 747, 2 (2012).
  * Hampel  _et al._ [2016] M. Hampel, R. J. Stancliffe, M. Lugaro, and B. S. Meyer, The intermediate neutron-capture process and carbon-enhanced metal-poor stars, Astrophys. J. 831, 171 (2016).
  * Roederer  _et al._ [2014] I. U. Roederer, H. Schatz, J. E. Lawler, T. C. Beers, J. J. Cowan, A. Frebel, I. I. Ivans, C. Sneden, and J. S. Sobeck, New detections of arsenic, selenium, and other heavy elements in two metal-poor stars, Astrophys. J. 791, 32 (2014).
  * Peterson  _et al._ [2020] R. Peterson, B. Barbuy, and M. Spite, Trans-iron Ge, As, Se, and heavier elements in the dwarf metal-poor stars HD 19445, HD 84937, HD 94028, HD 140283, and HD 160617, Astron. Astrophys. 638, A64 (2020).
  * Cowan and Rose [1977] J. J. Cowan and W. K. Rose, Production of 14C and neutrons in red giants., Astrophys. J. 212, 149 (1977).
  * Suda and Fujimoto [2010] T. Suda and M. Y. Fujimoto, Evolution of low-and intermediate-mass stars with [Fe/H]$\leq$\- 2.5, Monthly Notices of the Royal Astronomical Society 405, 177 (2010).
  * Cruz  _et al._ [2013] M. Cruz, A. Serenelli, and A. Weiss, S-process in extremely metal-poor, low-mass stars, Astron. Astrophys. 559, A4 (2013).
  * Denissenkov  _et al._ [2019] P. A. Denissenkov, F. Herwig, P. Woodward, R. Andrassy, M. Pignatari, and S. Jones, The i-process yields of rapidly accreting white dwarfs from multicycle he-shell flash stellar evolution models with mixing parametrizations from 3d hydrodynamics simulations, Monthly Notices of the Royal Astronomical Society 488, 4258 (2019).
  * Roederer  _et al._ [2016] I. U. Roederer, M. Mateo, J. I. Bailey III, Y. Song, E. F. Bell, J. D. Crane, S. Loebman, D. L. Nidever, E. W. Olszewski, S. A. Shectman, _et al._ , Detailed chemical abundances in the r-process-rich ultra-faint dwarf galaxy reticulum 2, Astron. J. 151, 82 (2016).
  * Choplin  _et al._ [2022] A. Choplin, S. Goriely, and L. Siess, Synthesis of thorium and uranium in asymptotic giant branch stars, Astron. Astrophys. 667, L13 (2022).
  * Choplin  _et al._ [2021] A. Choplin, L. Siess, and S. Goriely, The intermediate neutron capture process-i. development of the i-process in low-metallicity low-mass agb stars, Astron. Astrophys. 648, A119 (2021).
  * Schatz [2016] H. Schatz, Trends in nuclear astrophysics, J. Phys. G Nuc. Part. Phys. 43, 064001 (2016).
  * Han  _et al._ [2018] W. Han, L. Zhang, G. Yang, P. Niu, and B. Zhang, Study of the abundance features of the metal-poor star HD 94028, Astrophys. J. 856, 58 (2018).
  * Roederer  _et al._ [2022] I. U. Roederer, J. E. Lawler, E. A. D. Hartog, V. M. Placco, R. Surman, T. C. Beers, R. Ezzeddine, A. Frebel, T. T. Hansen, K. Hattori, E. M. Holmbeck, and C. M. Sakari, The r-process alliance: A nearly complete r-process abundance template derived from ultraviolet spectroscopy of the r-process-enhanced metal-poor star hd 222925*, Astrophys. J. Supplement Series 260, 27 (2022).
  * Dilling  _et al._ [2003] J. Dilling, P. Bricault, M. Smith, and H. J. Kluge, The proposed TITAN facility at ISAC for very precise mass measurements on highly charged short-lived isotopes, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 204, 492 (2003).
  * Ball  _et al._ [2011] G. C. Ball, L. Buchmann, B. Davids, R. Kanungo, C. Ruiz, and C. E. Svensson, Physics with reaccelerated radioactive beams at TRIUMF-ISAC, J. Phys. G Nuc. Part. Phys. 38, 10.1088/0954-3899/38/2/024003 (2011).
  * Egoriti  _et al._ [2018] L. Egoriti, R. Augusto, P. Bricault, T. D. Goodacre, M. Delonca, M. Dierckx, D. Houngbo, L. Popescu, J. Ramos, S. Rothe, T. Stora, and A. Gottberg, Development of a proton-to-neutron converter for radioisotope production at ISAC-TRIUMF, Journal of Physics: Conference Series 1067, 082022 (2018).
  * Lassen  _et al._ [2017] J. Lassen, R. Li, S. Raeder, X. Zhao, T. Dekker, H. Heggen, P. Kunz, C. D. P. Levy, M. Mostamand, A. Teigelhöfer, and F. Ames, Current developments with TRIUMF’s titanium-sapphire laser based resonance ionization laser ion source, Hyperfine Interact. 238, 33 (2017).
  * Dombsky  _et al._ [2000] M. Dombsky, D. Bishop, P. Bricault, D. Dale, A. Hurst, K. Jayamanna, R. Keitel, M. Olivo, P. Schmor, and G. Stanford, Commissioning and initial operation of a radioactive beam ion source at isac, Review of Scientific Instruments 71, 978 (2000), https://doi.org/10.1063/1.1150364 .
  * Bricault  _et al._ [2002] P. Bricault, R. Baartman, M. Dombsky, A. Hurst, C. Mark, G. Stanford, and P. Schmor, TRIUMF-ISAC target station and mass separator commissioning, Nuclear Physics A 701, 49 (2002).
  * Jesch  _et al._ [2015] C. Jesch, T. Dickel, W. R. Plaß, D. Short, S. A. San Andres, J. Dilling, H. Geissel, F. Greiner, J. Lang, K. G. Leach, W. Lippert, C. Scheidenberger, and M. I. Yavor, The MR-TOF-MS isobar separator for the TITAN facility at TRIUMF, Hyperfine Interact. 235, 97 (2015).
  * Reiter  _et al._ [2021] M. Reiter, S. A. S. Andrés, J. Bergmann, T. Dickel, J. Dilling, A. Jacobs, A. Kwiatkowski, W. Plaß, C. Scheidenberger, D. Short, C. Will, C. Babcock, E. Dunling, A. Finlay, C. Hornung, C. Jesch, R. Klawitter, B. Kootte, D. Lascar, E. Leistenschneider, T. Murböck, S. Paul, and M. Yavor, Commissioning and performance of titan’s multiple-reflection time-of-flight mass-spectrometer and isobar separator, Nucl. Instruments Methods Phys. Res. Sect. A Accel.Spectrometers, Detect. Assoc. Equip. 1018, 165823 (2021).
  * Plaß  _et al._ [2013] W. R. Plaß, T. Dickel, and C. Scheidenberger, Multiple-reflection time-of-flight mass spectrometry, Int. J. Mass Spectrom. 349-350, 134 (2013), 100 years of Mass Spectrometry.
  * Dickel  _et al._ [2015] T. Dickel, W. Plaß, A. Becker, U. Czok, H. Geissel, E. Haettner, C. Jesch, W. Kinsel, M. Petrick, C. Scheidenberger, A. Simon, and M. Yavor, A high-performance multiple-reflection time-of-flight mass spectrometer and isobar separator for the research with exotic nuclei, Nucl. Instruments Methods Phys. Res. Sect. A Accel.Spectrometers, Detect. Assoc. Equip. 777, 172 (2015).
  * Dickel  _et al._ [2017a] T. Dickel, M. I. Yavor, J. Lang, W. R. Plaß, W. Lippert, H. Geissel, and C. Scheidenberger, Dynamical time focus shift in multiple-reflection time-of-flight mass spectrometers, Int. J. Mass Spectrom. 412, 1 (2017a).
  * Leistenschneider  _et al._ [2018] E. Leistenschneider, M. Reiter, S. A. San Andrés, B. Kootte, J. Holt, P. Navrátil, C. Babcock, C. Barbieri, B. Barquest, J. Bergmann, _et al._ , Dawning of the N = 32 shell closure seen through precision mass measurements of neutron-rich titanium isotopes, Phys. Rev. Lett. 120, 062503 (2018).
  * Dickel  _et al._ [2017b] T. Dickel, W. R. Plaß, W. Lippert, J. Lang, M. I. Yavor, H. Geissel, and C. Scheidenberger, Isobar separation in a multiple-reflection time-of-flight mass spectrometer by mass-selective re-trapping, Journal of The American Society for Mass Spectrometry 28, 1079 (2017b).
  * Jacobs [aion] A. Jacobs, Nucl. Instruments Methods Phys. Res. Sect. A Accel.Spectrometers, Detect. Assoc. Equip. (2023, in prepartaion).
  * Beck  _et al._ [2021] S. Beck, B. Kootte, I. Dedes, T. Dickel, A. A. Kwiatkowski, E. M. Lykiardopoulou, W. R. Plaß, M. P. Reiter, C. Andreoiu, J. Bergmann, T. Brunner, D. Curien, J. Dilling, J. Dudek, E. Dunling, J. Flowerdew, A. Gaamouci, L. Graham, G. Gwinner, A. Jacobs, R. Klawitter, Y. Lan, E. Leistenschneider, N. Minkov, V. Monier, I. Mukul, S. F. Paul, C. Scheidenberger, R. I. Thompson, J. L. Tracy, M. Vansteenkiste, H.-L. Wang, M. E. Wieser, C. Will, and J. Yang, Mass measurements of neutron-deficient yb isotopes and nuclear structure at the extreme proton-rich side of the ${N}=82$ shell, Phys. Rev. Lett. 127, 112501 (2021).
  * Mukul  _et al._ [2021] I. Mukul, C. Andreoiu, J. Bergmann, M. Brodeur, T. Brunner, K. A. Dietrich, T. Dickel, I. Dillmann, E. Dunling, D. Fusco, G. Gwinner, C. Izzo, A. Jacobs, B. Kootte, Y. Lan, E. Leistenschneider, E. M. Lykiardopoulou, S. F. Paul, M. P. Reiter, J. L. Tracy, J. Dilling, and A. A. Kwiatkowski, Examining the nuclear mass surface of rb and sr isotopes in the $a\approx 104$ region via precision mass measurements, Phys. Rev. C 103, 044320 (2021).
  * Izzo  _et al._ [2021] C. Izzo, J. Bergmann, K. A. Dietrich, E. Dunling, D. Fusco, A. Jacobs, B. Kootte, G. Kripkó-Koncz, Y. Lan, E. Leistenschneider, E. M. Lykiardopoulou, I. Mukul, S. F. Paul, M. P. Reiter, J. L. Tracy, C. Andreoiu, T. Brunner, T. Dickel, J. Dilling, I. Dillmann, G. Gwinner, D. Lascar, K. G. Leach, W. R. Plaß, C. Scheidenberger, M. E. Wieser, and A. A. Kwiatkowski, Mass measurements of neutron-rich indium isotopes for $r$-process studies, Phys. Rev. C 103, 025811 (2021).
  * Porter  _et al._ [2022] W. S. Porter, B. Ashrafkhani, J. Bergmann, C. Brown, T. Brunner, J. D. Cardona, D. Curien, I. Dedes, T. Dickel, J. Dudek, E. Dunling, G. Gwinner, Z. Hockenbery, J. D. Holt, C. Hornung, C. Izzo, A. Jacobs, A. Javaji, B. Kootte, G. Kripkó-Koncz, E. M. Lykiardopoulou, T. Miyagi, I. Mukul, T. Murböck, W. R. Plaß, M. P. Reiter, J. Ringuette, C. Scheidenberger, R. Silwal, C. Walls, H. L. Wang, Y. Wang, J. Yang, J. Dilling, and A. A. Kwiatkowski, Mapping the ${N}=40$ island of inversion: Precision mass measurements of neutron-rich fe isotopes, Phys. Rev. C 105, L041301 (2022).
  * Ayet San Andrés  _et al._ [2019] S. Ayet San Andrés, C. Hornung, J. Ebert, W. R. Plaß, T. Dickel, H. Geissel, C. Scheidenberger, J. Bergmann, F. Greiner, E. Haettner, C. Jesch, W. Lippert, I. Mardor, I. Miskun, Z. Patyk, S. Pietri, A. Pihktelev, S. Purushothaman, M. P. Reiter, A. K. Rink, H. Weick, M. I. Yavor, S. Bagchi, V. Charviakova, P. Constantin, M. Diwisch, A. Finlay, S. Kaur, R. Knöbel, J. Lang, B. Mei, I. D. Moore, J. H. Otto, I. Pohjalainen, A. Prochazka, C. Rappold, M. Takechi, Y. K. Tanaka, J. S. Winfield, and X. Xu, High-resolution, accurate multiple-reflection time-of-flight mass spectrometry for short-lived, exotic nuclei of a few events in their ground and low-lying isomeric states, Phys. Rev. C 99, 10.1103/PhysRevC.99.064313 (2019).
  * Paul  _et al._ [2021] S. F. Paul, J. Bergmann, J. D. Cardona, K. A. Dietrich, E. Dunling, Z. Hockenbery, C. Hornung, C. Izzo, A. Jacobs, A. Javaji, B. Kootte, Y. Lan, E. Leistenschneider, E. M. Lykiardopoulou, I. Mukul, T. Murböck, W. S. Porter, R. Silwal, M. B. Smith, J. Ringuette, T. Brunner, T. Dickel, I. Dillmann, G. Gwinner, M. MacCormick, M. P. Reiter, H. Schatz, N. A. Smirnova, J. Dilling, and A. A. Kwiatkowski, Mass measurements of ${}^{60-63}\mathrm{Ga}$ reduce x-ray burst model uncertainties and extend the evaluated $T=1$ isobaric multiplet mass equation, Phys. Rev. C 104, 065803 (2021).
  * Paul [2020] S. F. Paul, emgfit - Fitting of time-of-flight mass spectra with hyper-EMG models (2020).
  * Purushothaman  _et al._ [2017] S. Purushothaman, S. Ayet San Andrés, J. Bergmann, T. Dickel, J. Ebert, H. Geissel, C. Hornung, W. Plaß, C. Rappold, C. Scheidenberger, Y. Tanaka, and M. Yavor, Hyper-EMG: A new probability distribution function composed of exponentially modified gaussian distributions to analyze asymmetric peak shapes in high-resolution time-of-flight mass spectrometry, Int. J. Mass Spectrom. 421, 245 (2017).
  * Wang  _et al._ [2021] M. Wang, W. J. Huang, F. G. Kondev, G. Audi, and S. Naimi, The AME 2020 atomic mass evaluation (II). Tables, graphs and references, Chinese Physics C 45, 10.1088/1674-1137/abddaf (2021).
  * Reiter  _et al._ [2020] M. Reiter, S. A. San Andrés, S. Nikas, J. Lippuner, C. Andreoiu, C. Babcock, B. Barquest, J. Bollig, T. Brunner, T. Dickel, _et al._ , Mass measurements of neutron-rich gallium isotopes refine production of nuclei of the first r-process abundance peak in neutron-star merger calculations, Phys. Rev. C 101, 025803 (2020).
  * Koning  _et al._ [2008] A. J. Koning, S. Hilaire, and M. C. Duijvestijn, Talys-1.0, Proceedings of the International Conference on Nuclear Data for Science and Technology, (2008).
  * Möller  _et al._ [2016] P. Möller, A. Sierk, T. Ichikawa, and H. Sagawa, Nuclear ground-state masses and deformations: FRDM(2012), Atomic Data and Nuclear Data Tables 109-110, 1 (2016).
  * Koning and Delaroche [2003] A. Koning and J. Delaroche, Local and global nucleon optical models from 1 keV to 200 MeV, Nuclear Physics A 713, 231–310 (2003).
  * Dilg  _et al._ [1973] W. Dilg, W. Schantl, H. Vonach, and M. Uhl, Level density parameters for the back-shifted fermi gas model in the mass range 40$<$ A $<$ 250, Nuclear Physics A 217, 269–298 (1973).
  * Gilbert and Cameron [1965] A. Gilbert and A. G. W. Cameron, A composite nuclear-level density formula with shell corrections, Canadian Journal of Physics 43, 1446–1496 (1965), http://dx.doi.org/10.1139/p65-139 .
  * Kopecky and M.Uhl [1990] J. Kopecky and M.Uhl, Test of gamma-ray strength functions in nuclear reaction model calculations, Phys. Rev. C 41, 1941 (1990).
  * Mendoza-Temis  _et al._ [2015] J. d. J. Mendoza-Temis, M.-R. Wu, K. Langanke, G. Martínez-Pinedo, A. Bauswein, and H.-T. Janka, Nuclear robustness of the $r$ process in neutron-star mergers, Phys. Rev. C 92, 055805 (2015).
  * Lippuner and Roberts [2015] J. Lippuner and L. F. Roberts, r-process lanthanide production and heating rates in kilonovae, Astrophys. J. 815, 82 (2015).
  * Goriely [1999] S. Goriely, Uncertainties in the solar system r-abundance distribution, Astron. Astrophys. 342, 881 (1999).
  * Wanajo [2018] S. Wanajo, Physical conditions for the r-process. I. radioactive energy sources of kilonovae, Astrophys. J. 868, 65 (2018).
  * Wu  _et al._ [2019] M.-R. Wu, J. Barnes, G. Martínez-Pinedo, and B. D. Metzger, Fingerprints of heavy-element nucleosynthesis in the late-time lightcurves of kilonovae, Phys. Rev. Lett. 122, 062701 (2019).
  * Roederer  _et al._ [2018] I. U. Roederer, C. M. Sakari, V. M. Placco, T. C. Beers, R. Ezzeddine, A. Frebel, and T. T. Hansen, The r-process alliance: a comprehensive abundance analysis of HD 222925, a metal-poor star with an extreme r-process enhancement of [Eu/H]= -0.14, Astrophys. J. 865, 129 (2018).
  * Ji  _et al._ [2016] A. P. Ji, A. Frebel, A. Chiti, and J. D. Simon, R-process enrichment from a single event in an ancient dwarf galaxy, Nature 531, 610 (2016).


# Joint Deep Learning for Car Detection

Seyedshams Feyzabadi
Electrical Engineering and Computer Science
University of California, Merced
<EMAIL_ADDRESS>
###### Abstract

Traditional object recognition approaches apply feature extraction, part
deformation handling, occlusion handling and classification sequentially while
they are independent from each other. Ouyang and Wang proposed a model for
jointly learning of all of the mentioned processes using one deep neural
network. We utilized, and manipulated their toolbox in order to apply it in
car detection scenarios where it had not been tested. Creating a single deep
architecture from these components, improves the interaction between them and
can enhance the performance of the whole system. We believe that the approach
can be used as a general purpose object detection toolbox. We tested the
algorithm on UIUC car dataset, and achieved an outstanding result. The
accuracy of our method was 97 % while the previously reported results showed
an accuracy of up to 91 %. We strongly believe that having an experiment on a
larger dataset can show the advantage of using deep models over shallow ones.

##  1 Introduction

Object recognition plays an important role in computer vision, and has been
extensively discussed in the literature [20, 5, 26, 21]. The main difficulty
in creating a robust object detection approach comes from the wide range of
variations in images of objects belonging to the same object class. Similar
objects from the same class might have different sizes, different shapes
because of the view points etc. Having a part-based model helps to overcome
these issues and create an abstract model of the object by using its parts.
Similar challenges of different textures, lighting, background and etc.
increase the difficulties of detecting an object in a picture.

The traditional approach of overcoming these challenges follows the following
steps:

  1. 1.

Feature extraction: features are extracted from an image to find out the most
discriminative parts of it. The most famous approaches are SIFT [22] or Haar
[27]

  2. 2.

Deformation handling: extracting multiple parts of an object will be very
helpful when there are variety of shapes for the same object while they
consist of similar part. For instance, cars have different shapes, but they
all consist of body, tires, trunk and hood. Agarwal performed a good research
on part-based object detection [1, 2].

  3. 3.

Occlusion handling: Occluded parts have to be detected in order to avoid using
them in object detection itself. Including them in the detection phase will
reduce the accuracy of the program. Some usages of occlusion detection
algorithms can be found here [28, 8].

  4. 4.

Classification: After finishing all the procedures, a classifier decides
whether the chosen window contains an instance of that object or no. The most
known approaches are SVM [6] and random forests [7].

Most of the traditional approaches follow these steps sequentially. Each of
them is created and trained first, and the output of each step is fed as the
input to the next step. However, the interactions and feedbacks of each step
to the other one has not been studied well.

Ouyang et al [24] proposed the first model for joint learning of all of these
steps in one single deep convolutional network and reported good results in
pedestrian detection scenarios. Their model is based on learning all of the
steps at the same time, and use the output of higher levels in training the
lower levels. Figure 1 shows their approach for joint deep learning in
pedestrian detection. Their method is capable of using the training data in
creating better low level filters. This vividly explains why the joint
learning can be beneficial comparing to independent learning.

In this paper we focus on Ouyang’s work [24] for pedestrian detection and
mostly base our work on their efforts. The main contribution of this paper is
to generalize their framework and apply it on other applications. The author
strongly believes that this approach has the potential to become a generic
framework for object recognition. The biggest advantage of this approach is
its flexibilities where it makes the approach very easy to change application.
We can imagine that the same method can be applied to other objects too. We
also believe that the deformation layer of this methodology makes it more
useful for objects that are composed of many parts. As an example a bicycle
can be a good representative object. Another interesting application can be in
animal detection where their body parts are not rigid and may have different
articulations.

Figure 1: Joint deep learning for pedestrian detection, taken from [24]

##  2 Related Work

Several approaches to object detection were proposed in the past that use some
form of learning. In most such approaches, images are represented by using
some features, and a learning method is used to identify regions in the
feature space that correspond to the object class. There are a large variety
in types of features used and the learning methods applied [11].

Recently deep models showed great performance in machine learning and
recognition problems [18, 14, 17, 19, 25, 9] comparing to shallow approaches.
There are some robotics usages of deep and shallow models too[13, 10, 4]. They
proved and showed that these approaches are capable of solving complex tasks.
Their complicated structure makes them more compatible for complex problems
with a lot of details. But, it arises the problem of unknown regions. There is
not any proof why they are so efficient [18], how they can be improved or what
is a structured way of creating the best model.

Most of the traditional approaches use features such as SIFT or HOG [6] to
extract the overall shape of an object, and apply different learning
approaches to train the system. All of these approaches have one common point
that the features are manually generated, and only used for the training
purposes. While some recent investigations show that learning features from
the training data is very helpful and improves the accuracy of the program
[3].

Detecting cars is an important aspect in traffic control scenarios. It will be
very useful to distinguish cars from other objects. Multiple approaches have
focused on this problem [12, 1, 2]. Since different types of cars have
different shapes, but are composed from similar parts, it will be very
beneficial to create a part-based model for it. Then the same solver can be
applied to different classes of cars such as sedans, SUVs, trucks and etc.
Comparing to other objects, cars have very huge variety of shapes and sizes.
Thus it will be too hard, if not impossible, to have a general model for all
of them. But in this work, we only focus on sedan cars.

We propose a model for car detection that benefits from deep learning
approaches and is capable of detecting different classes of cars. The approach
uses the training data to improve its low-level features. Even though we
applied this approach to side view pictures of cars, it is extendable to other
views of the cars too. We very closely followed the work by Ouyang [24] and
used the same methodology to detect different object class.

##  3 Our Approach

As discussed earlier, the main difference between this method with traditional
recognition method is the jointly learning of the mentioned processes. Its
architecture is shown in Figure 2. In general, the following steps are
performed in this program:

  * •

The first convolutional layer generates filtered data maps. This layer
convolves the 3-channel input image data with 9 x 9 x 3 filters and outputs 64
maps.

  * •

Average pooling is applied to Features maps with the size of 64 filtered data
maps using 4 x 4 boxcar filters with a 4 x 4 subsampling step.

  * •

The second convolutional layer generates part detection maps. This layer
convolves the feature maps with 8 part filters of different sizes and outputs
8 part detection maps.

  * •

Part scores are obtained from the 8 part detection maps using a deformation
handling layer. This layer outputs 8 part scores.

  * •

The visibility reasoning of 8 parts is used for estimating the label y; that
is, whether a given window encloses a car or not.

Figure 2: Program architecture, taken from [24]

###  3.1 Input Data

The size of an input image is assumed to be 84x28 and it is fed to the program
with the following three channels:

  * •

The first contains Y data in YUV format. Its size is 84 x 28.

  * •

The three-channel 42 x 14 images in the YUV color space are appended into the
second channel of size 84 x 28 with zero padding.

  * •

Four 42 x 14 edge maps are appended into the third channel of size 84x28.
Three edge maps are obtained from the three-channel images in the YUV color
space. The magnitudes of horizontal and vertical edges are computed using the
Sobel edge detector [15]. The fourth edge map is chosen as the maximum
magnitudes from the first three edge maps.

Ouyang et al, found it useful to feed the program in three channels as
explained above, and we follow their strategy. They claimed that

> In this way, information about pixel values at different resolutions and
> information of primitive edges are utilized as the input of the first
> convlutional layer to extract features. The first convolutional layer and
> its following average pooling layer use the standard CNN settings. We
> empirically find that it is better to arrange the images and edge maps into
> three concatenated channels instead of eight separate channels. In order to
> deal with illumination change, the data in each channel is preprocessed to
> be zero mean and unit variance [24].

###  3.2 Part Maps

Most of the vision approaches have filters with equal sizes, but cars might
have different sizes or been seen from different distances.. Therefore, we
decided to have variable size filter as shown in Figure 3. The filters are
defined on three levels and the size of the largest one is 5 x 15. The smaller
ones have portion of the whole filter. The side view image of a car can be
taken from both sides, therefore, it is needed to have filters for both views.
The left and right side of Figure 3 deal with this issue.

Figure 3: Part filters for the side view pictures of cars.

###  3.3 Deformation Layer

In this subsection we would like to explain what happens in the second
convolutional layer. This layer receives the P part detection maps as input
and part scores s = {$s_{1},...,s_{P}$}, P = 8, are its output. The
convolutional layer treats the detection maps individually and produces the
pth part score $s_{p}$ from the pth part detection map, denoted by $M_{p}$. A
2D summation map, $B_{p}$, is calculated by summing up the part detection map
$M_{p}$ and the deformation maps as follows:

$B_{p}=M_{p}+\Sigma c_{n,p}D_{n,p}$

$D{n,p}$ denotes the nth deformation map for the pth part, $c_{n,p}$ denotes
the weight for $D_{n,p}$, and N denotes the number of deformation maps. sp is
globally max-pooled from $B_{p}$

$s_{p}=maxb_{p}^{(x,y)}$

where $b_{p}^{(x,y)}$ denotes the (x, y)th element of $B_{p}$. The detected
part location can be inferred from the summed map as follows:

$(x,y)_{p}=argmaxb_{p}^{(x,y)}$

At the training stage, only the value at location $(x,y)_{p}$ of $B_{p}$ is
used for learning the deformation parameters.

The $c_{n,p}$ and $D_{n,p}$ are the key for designing different deformation
models. Both $c_{n,p}$ and $D_{n,p}$ can be considered as the parameters to be
learned. Three examples are given below:

  1. 1.

Suppose N = 1, $c_{1,p}$ = 1 and the deformation map $D_{1,p}$ is to be
learned. In this case, the discrete locations of the pth part are treated as
bins and the deformation cost for each bin is learned. $d_{1,p}^{(x,y)}$,
which denotes the (x, y)th element of D1,p, corresponds to the deformation
cost of the pth part at location (x, y).

  2. 2.

$D_{1,p}$ can also be predefined. Suppose N = 1 and $c_{n,p}$ = 1. If
$d_{1,p}^{(x,y)}$ is the same for any (x, y), then there is no deformation
cost. If $d_{1,p}^{(x,y)}=-\infty$ for $(x,y)\in X$, $d_{1,p}^{(x,y)}=0$ for
$(x,y)\in X$, then the parts are only allowed to move freely in the location
set X. Max-pooling is a special case of this example by setting X to be a
local region. The disadvantage of max-pooling is that the hand-tuned local
region does not adapt to different deformation properties of different parts.

  3. 3.

The deformation layer can represent the widely used quadratic constraint of
deformation. Below, we skip the subscript p. The quadratic constraint of
deformation can be represented as follows:

$b^{(x,y)}=m^{(x,y)}+c_{1}(x-a_{x}+\frac{c_{3}}{2c_{1}})^{2}+c_{2}(y-a_{y}+\frac{c_{4}}{2c_{2}})^{2}$
(1)

where $m^{(x,y)}$ is the (x, y)th element of the part detection map M,
$(a_{x},a_{y})$ is the predefined anchor location of the pth part. They are
adjusted by c3/2c1 and c4/2c2, which are automatically learned. c1 and c2
decide the deformation cost. There is no deformation cost if c1 = c2 = 0.
Parts are not allowed to move if $c1=c2=−\infty$. (ax, ay) and
$(\frac{c_{3}}{2c_{1}},\frac{c_{4}}{2c_{2}})$ jointly decide the center of the
part. The quadratic constraint can be represented as follows:

$\displaystyle B=M+$ $\displaystyle
c_{1}D_{1}+c_{2}D_{2}+c_{3}D3+c_{4}D_{4}+c_{5}.1,$ (2) $\displaystyle
b^{(x,y)}=$ $\displaystyle
m^{(x,y)}+c_{1}d_{1}^{(x,y)}+c_{2}d_{2}^{(x,y)}+c_{3}d_{3}^{(x,y)}$
$\displaystyle+c_{4}d_{4}^{(x,y)}+c_{5},$ $\displaystyle d_{1}^{(x,y)}=$
$\displaystyle(x-a_{x})^{2},$ $\displaystyle d_{2}^{(x,y)}=$
$\displaystyle(y-a_{y})^{2},$ $\displaystyle d_{3}^{(x,y)}=$ $\displaystyle
x-a_{x},$ $\displaystyle d_{4}^{(x,y)}=$ $\displaystyle y-a_{y},$
$\displaystyle c_{5}=$
$\displaystyle\frac{c_{3}^{2}}{4c_{1}}+\frac{c_{4}^{2}}{4c_{2}}$

where $1$ is a matrix with all elements being one, $d_{n}^{(x,y)}$ is the (x,
y)th element of $D_{n}$. In this case, c1, c2, c3 and c4 are parameters to be
learned and $D_{n}$ are predefined. c5 is the same in all locations and need
not be learned.

Figure 4: The deformation layer. Part detection map and deformation maps are
summed up with weights $c_{n,p}$ for n = 1, 2, 3, 4 to obtain the summed map
$B_{p}$. Global max pooling is then performed on the summed map to obtain the
score sp for the pth part. Taken from [23].

Figure 4 is an example of learning these parameters. In this case parameters
$c_{1}$, $c_{2}$, $c_{3}$ and $c_{4}$ are to be learned and $D_{n}$ is
predefined. It is notable that a large part of this subsection follows the
work of Ouyang.

###  3.4 Classification

The results of previous section is a set of scores $s_{1},s_{2},...,s_{n}$.
Then visibility reasoning is used for classification. More details of it can
be found in [23].

In this paper, features, deformable models, and visibility relationships are
jointly learned. Backpropagation through $s$ is used as prediction in order to
learn the parameters in the two convolutional layers and the deformation layer
in Figure 2.

In order to train this deep architecture, we used a multi-stage training
strategy. We start with a 1-layer CNN using supervised training. Since Gabor
filters [16] are similar to the human visual system, they are used for
initializing the first CNN. We add one more layer at each stage, the layers
trained in the previous stage are used for initialization and then all the
layers at the current stage are jointly optimized with back propagation.

##  4 Experimental Results

The described architecture is tested on UIUC database
111http://cogcomp.cs.illinois.edu/Data/Car/. The database contains 1050
images(550 cars and 500 non-cars). The test set consists of 170 images with
200 cars in them. The file format of the images are PGM. There are some
evaluation codes in the dataset which we did not use for this experiment, but
they can be utilized for evaluating different object recognition algorithms. A
snapshot of the data set is shown in Figure 5.

Image sizes are 40 x 100 and they are in gray scale. In order to be in line
with Ouyang’s work, we rotated and scaled the images to 84X28. Therefore, we
can use the same set of filter as them.

We used log average miss rate in order to make comparisons with other methods.
It is computed by averaging the miss rate at nine FPPI rates that are evenly
spaced in the log-space in the range from 0.01 to 1.

In order to improve the results we applied augmentation. This is done by
creating manual rotations on the images starting from -10 degrees up to +10
with intervals of 1 degree.

Figure 5: UIUC dataset. The upper part shows the positive images, and the
lower part is the negative samples.

As a comparison method we picked the method developed by Agarwal et al [1].
They used the same data set with similar test images. Their method is robust
and accurate and a sample results of their method is shown in Figure 6. Their
average miss rate is 9 %.

Figure 6: The results of Agarwal’s work, published in [1].

The results of our experiments without augmentation is shown in Figure 7.
However, the results is not disappointing. It is not too bad either. The miss
rate of this approach is 23 %.

Figure 7: Jointly learning without Augmentation.

Augmentation is needed to improve the dataset and makes it more stable against
camera angles. After we applied the augmentation, we achieved the results in
Figure 8. This method decreased the average miss rate to 3 % which shows a
great improvement comparing to the reported 9 % from Agarwal’s work.

Figure 8: The results of our algorithm after applying augmentation.

##  5 Conclusions

In this report we showed our work about using joint deep learning architecture
for car detection applications. The main idea of joint deep learning is to
include feature extraction, part deformation handling, occlusion handling and
classification in one single deep neural network where two layers of
convolutional layers exist. The first layer extracts feature maps using low-
level features that are tuned during the training phase. And the second
convolutional layer is responsible for deformation handling to extract a score
based on different visible parts of an object.

We applied the algorithm on UIUC car dataset, and managed to gain good results
of up to 97 % of success rate, after applying augmentation. Comparing our
results with older results of Agarwal, our method outperforms the Agarwal’s
result by 6%. We also showed the importance of augmentation in the final
performance of an algorithm. Since deep learning approaches require large
datasets, we increased the size of our database by applying augmentation.

## References

  * [1] S. Agarwal, A. Awan, and D. Roth.  Learning to detect objects in images via a sparse, part-based representation.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 26(11):1475–1490, 2004.
  * [2] S. Agarwal and D. Roth.  Learning a sparse representation for object detection.  In Computer Vision ECCV 2002, pages 113–127. Springer, 2002.
  * [3] A. Bar-Hillel, D. Levi, E. Krupka, and C. Goldberg.  Part-based feature synthesis for human detection.  In Computer Vision–ECCV 2010, pages 127–142. Springer, 2010.
  * [4] P. Baumann, S. Feyzabadi, and C. Jucovschi.  Putting pixels in place: A storage layout language for scientific data.  In Data Mining Workshops (ICDMW), 2010 IEEE International Conference on, pages 194–201. IEEE, 2010.
  * [5] S. Belongie, J. Malik, and J. Puzicha.  Shape matching and object recognition using shape contexts.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 24(4):509–522, 2002.
  * [6] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 886–893. IEEE, 2005.
  * [7] P. Dollár, R. Appel, and W. Kienzle.  Crosstalk cascades for frame-rate pedestrian detection.  In Computer Vision–ECCV 2012, pages 645–659. Springer, 2012.
  * [8] M. Enzweiler, A. Eigenstetter, B. Schiele, and D. M. Gavrila.  Multi-cue pedestrian classification with partial occlusion handling.  In Computer vision and pattern recognition (CVPR), 2010 IEEE Conference on, pages 990–997. IEEE, 2010.
  * [9] A. Ess, B. Leibe, and L. Van Gool.  Depth and appearance for mobile scene analysis.  In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pages 1–8. IEEE, 2007.
  * [10] S. Feyzabadi, S. Straube, M. Folgheraiter, E. A. Kirchner, S. K. Kim, and J. C. Albiez.  Human force discrimination during active arm motion for force feedback design.  Haptics, IEEE Transactions on, 6(3):309–319, 2013.
  * [11] D. A. Forsyth and J. Ponce.  Computer vision: a modern approach.  Prentice Hall Professional Technical Reference, 2002.
  * [12] C. Goerick, D. Noll, and M. Werner.  Artificial neural networks in real-time car detection and tracking applications.  Pattern Recognition Letters, 17(4):335–343, 1996.
  * [13] S. H. Hamraz and S. S. Feyzabadi.  General-purpose learning machine using k-nearest neighbors algorithm.  In RoboCup 2005: Robot Soccer World Cup IX, pages 529–536. Springer, 2006.
  * [14] G. Hinton, S. Osindero, and Y.-W. Teh.  A fast learning algorithm for deep belief nets.  Neural computation, 18(7):1527–1554, 2006.
  * [15] B. Jähne, H. Scharr, and S. Körkel.  Principles of filter design.  Handbook of computer vision and applications, 2:125–151, 1999.
  * [16] A. K. Jain and F. Farrokhnia.  Unsupervised texture segmentation using gabor filters.  In Systems, Man and Cybernetics, 1990. Conference Proceedings., IEEE International Conference on, pages 14–19. IEEE, 1990.
  * [17] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun.  What is the best multi-stage architecture for object recognition?  In Computer Vision, 2009 IEEE 12th International Conference on, pages 2146–2153. IEEE, 2009.
  * [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [19] Q. V. Le.  Building high-level features using large scale unsupervised learning.  In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8595–8598. IEEE, 2013.
  * [20] D. G. Lowe.  Object recognition from local scale-invariant features.  In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, volume 2, pages 1150–1157. Ieee, 1999.
  * [21] A. Mohan, C. Papageorgiou, and T. Poggio.  Example-based object detection in images by components.  Pattern Analysis and Machine Intelligence, IEEE Transactions on, 23(4):349–361, 2001.
  * [22] P. C. Ng and S. Henikoff.  Sift: Predicting amino acid changes that affect protein function.  Nucleic acids research, 31(13):3812–3814, 2003.
  * [23] W. Ouyang and X. Wang.  A discriminative deep model for pedestrian detection with occlusion handling.  In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3258–3265. IEEE, 2012.
  * [24] W. Ouyang and X. Wang.  Joint deep learning for pedestrian detection.  In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 2056–2063. IEEE, 2013.
  * [25] M. Ranzato, F. J. Huang, Y.-L. Boureau, and Y. LeCun.  Unsupervised learning of invariant feature hierarchies with applications to object recognition.  In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pages 1–8. IEEE, 2007.
  * [26] M. Riesenhuber and T. Poggio.  Hierarchical models of object recognition in cortex.  Nature neuroscience, 2(11):1019–1025, 1999.
  * [27] P. Viola, M. J. Jones, and D. Snow.  Detecting pedestrians using patterns of motion and appearance.  In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, pages 734–741. IEEE, 2003.
  * [28] X. Wang, T. X. Han, and S. Yan.  An hog-lbp human detector with partial occlusion handling.  In Computer Vision, 2009 IEEE 12th International Conference on, pages 32–39. IEEE, 2009.