# Mitigating Inappropriateness in Image Generation:
Can there be Value in Reflecting the World’s Ugliness?

Manuel Brack  Felix Friedrich  Patrick Schramowski  Kristian Kersting

###### Abstract

Text-conditioned image generation models have recently achieved astonishing
results in image quality and text alignment and are consequently employed in a
fast-growing number of applications. Since they are highly data-driven,
relying on billion-sized datasets randomly scraped from the web, they also
reproduce inappropriate human behavior. Specifically, we demonstrate
inappropriate degeneration on a large-scale for various generative text-to-
image models, thus motivating the need for monitoring and moderating them at
deployment. To this end, we evaluate mitigation strategies at inference to
suppress the generation of inappropriate content. Our findings show that we
can use models’ representations of the world’s ugliness to align them with
human preferences.

Machine Learning, ICML

Warning: This paper contains sexually explicit imagery, discussions of
pornography, and other content that some readers may find disturbing,
distressing, and/or offensive.

##  1 Introduction

Next to text-generative models such as ChatGPT, image-generative models are
becoming increasingly prevalent and seeing growing adoption in commercial
services such as stockimagery and graphic design. Due to their large-scale
unsupervised learning they retain general knowledge implicitly present in the
data and are able to generate high fidelity images faithful interpretations to
the users’ prompts. However, their learning setup, which includes large-scale
unfiltered data (schuhmann2022laion; birhane2021multimodal), also leads to
degenerated and biased behavior (schramowski2022safe), calling for mitigation
strategies and the moderation of generative models in deployed systems.

Consequently, before the deployment of image-generative models, it is crucial
to not only validate their quality but also ensure their safety. This
necessitates the assessment of appropriate guardrails, which should be
tailored to the specific application at hand. Previous work in this domain has
primarily relied on anecdotal evidence, lacking quantifiable measures that
take multiple models and architectures into account. Indeed,
schramowski2022safe proposed an empirical benchmark but limited their
evaluation to a single Stable Diffusion version.

To help the development of effective mitigation strategies and moderation
techniques for image-generative models in real-world systems, we here present
a comprehensive assessment of inappropriate degeneration across numerous open-
source models and architectures. More precisely, we investigate how
effectively these models can be instructed to suppress inappropriate content
using the knowledge obtained about the world’s ugliness. Our findings suggest
that safety mitigation of text-to-image generators can be performed through
direct instructions at inference for various types of models. In total, we
generated and evaluated over 1.5M images for 11 different models, thereby
providing a large-scale investigation of the topic.

Figure 1: Examples of inappropriate degeneration and their mitigation across
various models. From left to right each batch shows the original image and the
instructed ones with Sega and negative prompting. Prompts are taken from the
inappropriate-image-prompts (I2P) dataset. Images displaying nudity were
blurred by the authors. (Best viewed in color)

##  2 Instructing Models on the World’s Ugliness

Visual Moderation. There exist multiple approaches for mitigating
inappropriate degeneration of generative models. Previous research has
identified four major methods. The first approach involves filtering the
training data to remove problematic content entirely (nichol2022glide).
However, large-scale dataset filtering can have unexpected side effects on
downstream performance as demonstrated by nichol2022glide. Moreover,
determining what constitutes inappropriate content is highly subjective and
dependent on various external factors such as individual and societal norms as
well as the specific use case of the application. Developing a dedicated model
with data filtering tailored to each definition of inappropriateness is
difficult, if not impractical, particularly as it would require retraining
pre-existing models from scratch. To overcome this limitation, a second
approach involves finetuning a pre-trained model to erase inappropriate
concepts (gandikota2023erasing). While this method requires lower
computational resources compared to training an entire model, it is still
constrained in its ability to account for diverse definitions of
inappropriateness. Another relevant approach, particularly for deployed
applications, involves implementing input and output
filters111https://www.technologyreview.com/2023/02/24/1069093/. In hosted
inference services, input prompts are typically filtered for banned keywords,
and the generated images are scanned for inappropriate content before being
presented to users. Although this approach restricts the availability of
unwanted content, it has some drawbacks. schramowski2022safe have demonstrated
that inappropriate degeneration can occur unexpectedly for prompts lacking
explicit descriptions of any problematic concepts. Therefore, input filters
are prone to missing these implicit correlations. Additionally, the generation
and subsequent discarding of images not only wastes computational resources
but can also result in a frustrating user experience.

In contrast, we here explore the idea of leveraging a model’s learned
representations of inappropriate content for mitigation of such material. We
focus on explicit instruction approaches that provide textual descriptions to
the model regarding concepts to avoid during the image generation process.
This results in both high flexibility and customizability, as the instruction
prompt can be easily modified to adapt to different requirements.
Consequently, the user remains involved in the process and the method enables
seamless deployment across various architectures. As such they also facilitate
large-scale evaluation across models.

Classifier Free Guidance. Before going into detail on different instruction
methods for image generation, we need to establish some fundamentals of text-
to-image diffusion models (DMs). Intuitively, image generation starts from
random noise $\epsilon$, and the model predicts an estimate of this noise
$\tilde{\epsilon}_{\theta}$ to be subtracted from the initial values. This
results in a high-fidelity image $x$ without any noise. Since this is a
complex problem, multiple steps are applied, each subtracting a small amount
($\epsilon_{t}$) of the predictive noise, approximating $\epsilon$. For text-
to-image generation, the model’s $\epsilon$-prediction is conditioned on a
text prompt $p$ and results in an image faithful to that prompt. To that end,
DMs employ classifier-free guidance (ho2022classifier), a conditioning method
using a purely generational diffusion model, eliminating the need for an
additional pre-trained classifier. The noise estimate
$\tilde{\epsilon}_{\theta}$ uses an unconditioned $\epsilon$-prediction
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})$ which is pushed in the direction
of the conditioned estimate
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{p})$ to yield an image
faithful to prompt $p$.

Instructing Text-to-Image Models on Safety. We now consider two different
instruction approaches extending the principles of classifier-free guidance.
Both methods rely on a secondary text prompt $s$ that describes concepts to
suppress during generation. First, negative prompting replaces the
unconditioned $\epsilon$-prediction
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t})$ with one conditioned on $s$:
$\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{s})$, thus moving away
from the inappropriate concepts. This approach is intuitive and easy to
implement, however offers limited control over the extent of content
suppression. Additionally, we use Semantic Guidance (Sega) (brack2023Sega)
which is a powerful method for image manipulation based on additional text
prompts. Sega adds an additional guidance term to $\tilde{\epsilon}_{\theta}$
that allows us to steer the generation away from $s$, while keeping changes to
the original image minimal.

Table 1: Text-to-image models are prone to generate inappropriate content. Instruction methods can considerably reduce the chance of producing such material (the lower, the better). Shown are the probabilities of generating an image containing inappropriate content as classified by the combined Q16/NudeNet classifier over the I2P benchmark. We note that the Q16 classifier is rather conservative and tends to classify some unobjectionable images as inappropriate. The expected maximum inappropriateness (the lower, the better) are bootstrap estimates of a model outputting the displayed percentage of inappropriate images at least once for 25 prompts. Subscript values indicate the standard deviation. Models evaluated are Stable Diffusion (SD) (rombach2022High) and fine-tuned variants, AltDiffusion (chen2022altclip), MultiFusion (bellagente2023multifusion), Paella (rampas2023novel) and IF by Deepfloyd33footnotemark: 3.  | Base Model | w/ SEGA | w/ Neg. Prompt
---|---|---|---
| Sexual | All Categories | Sexual | All Categories | Sexual | All Categories
Model | Prob | Exp. Max | Prob | Exp. Max | Prob | Exp. Max | Prob | Exp. Max | Prob | Exp. Max | Prob | Exp. Max
SD 1.4 |  |  |  |  |  |  |  |  |  |  |  |


# Learning behavioral context recognition with multi-stream
temporal convolutional networks

Aaqib Saeed, Tanir Ozcelebi, *Stojan Trajanovski, Johan Lukkien
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Mathematics and Computer Science, Eindhoven University of
Technology, Eindhoven, The Netherlands
*Philips Research, Eindhoven, The Netherlands

###### Abstract

Smart devices of everyday use (such as smartphones and wearables) are
increasingly integrated with sensors that provide immense amounts of
information about a person’s daily life such as behavior and context. The
automatic and unobtrusive sensing of behavioral context can help develop
solutions for assisted living, fitness tracking, sleep monitoring, and several
other fields. Towards addressing this issue, we raise the question: can a
machine learn to recognize a diverse set of contexts and activities in a real-
life through joint learning from raw multi-modal signals (e.g. accelerometer,
gyroscope and audio etc.)? In this paper, we propose a multi-stream temporal
convolutional network to address the problem of multi-label behavioral context
recognition. A four-stream network architecture handles learning from each
modality with a contextualization module which incorporates extracted
representations to infer a user’s context. Our empirical evaluation suggests
that a deep convolutional network trained end-to-end achieves an optimal
recognition rate. Furthermore, the presented architecture can be extended to
include similar sensors for performance improvements and handles missing
modalities through multi-task learning without any manual feature engineering
on highly imbalanced and sparsely labeled dataset.

## Introduction

The problem of context recognition is centered on inferring person’s
environment, physical state, and activity performed at any particular time.
Specifically, a understanding of the user’s current context requires
determining where and with whom the person is? and in what type of activity
the person is involved in? The behavioral and activity analysis is an
important and challenging task mainly because it is crucial for several
applications, including smart homes (?), assisted living (?; ?), fitness
tracking (?), sleep monitoring (?), user-adaptive services, social interaction
(?) and in industry. In particular, an accurate recognition of human context
can greatly benefit healthcare and wellbeing through automatic monitoring and
supervision of patients with chronic diseases (?) such as hypertension,
diabetes and dementia (?). Furthermore, the gathered knowledge and extracted
activity patterns can enable novel treatment design, adjustment of
medications, better behavioral intervention and patient observation strategies
(?).

In practice, for a context detection system to be effective in a real-life
requires an unobtrusive monitoring. It is important to not distress a person
in order to capture their realistic behaviors in a natural environment. The
penetration of smart sensing devices (e.g. smartphones and wearables) that are
integrated with sophisticated sensors in our daily lives provides a great
opportunity to learn and infer about various aspects of a person’s daily life.
However, there is considerable variability in the human behavior in real-world
situations that can cause the system to fail, if it is developed using data
collected in a constrained environment. For instance, ? shows that the
accuracy of activity classification differs based on the interaction with the
phone e.g. when in hand or carried in the bag. The various sensors embedded in
the smart devices convey information about different ambient facets each with
a distinct prospect. The variability issues of different patterns in phone
usage, environments, and device types can be very well addressed (to improve
the recognition capability of the system) through learning disentangled
representations from a large-scale data source and fusing rich sensory
modalities rather than separately utilizing each of them.

Figure 1: Multi-modal representation learning from sensors: Schematic of the
proposed multi-stream convolutional network.

(a) Audio (MFCC) (b) Accelerometer (c) Gyroscope

Figure 2: Context recognition dataset: Samples from large-scale multi-modal
sensory data collected in-the-wild conditions. The individual plots within
each sub-figure correspond to the same set of activities/context.

In the past, several studies have shown great improvement in sensor processing
for basic activity recognition (?; ?). The majority of the earlier methods use
shallow learning classifiers (such as, Random Forest and Support Vector
Machine) with hand-engineered features extracted from raw sensor readings e.g.
heuristically selected statistical or frequency measures (?). Likewise, many
studies involve simulated controlled trials for data collection in lab
environments that require users to wear extra sensors. Broadly, they also
treat activity recognition as a multi-class classification problem, where a
user’s activity at a specific moment can be defined by one of the k defined
classes. On the contrary, people are not generally engaged in just one
activity in their day-to-day living e.g. a person might surf the web while
eating or talking to friends. These problems limit the applicability of these
studies to detect very few rudimentary activities and make it harder for the
system to generalize to real-life settings. Nevertheless, to be successful in
everyday scenarios, the context recognition module should support a diverse
set of activities, varying device usage, and a wide range of environments.
Importantly, it must not only learn discriminative representations directly
from raw signals without any ad-hoc feature engineering, but also seamlessly
combine the discovered explanatory factors in the milieu of diverse sensory
modalities (?).

In recent years, the fields of speech recognition, drug discovery, image
segmentation and machine translation have been tremendously revolutionized
thanks to the availability of massive labeled datasets and end-to-end deep
representation learning (?). Similarly, the domain of human activity
recognition has also started leveraging deep neural networks for automatic
feature learning (?; ?; ?) though commonly restricted to the detection of only
elementary activities such as, walking, sitting, standing etc. There has not
been the same progress in recognizing complex behavioral context in daily-life
situations using devices of daily use. This can be partially attributed to the
lack of a large labeled dataset, which is both expensive and time-consuming to
accumulate in a real-world settings. We believe that large-scale sensory data
can significantly advance context recognition. This issue is very recently
addressed in (?; ?) which has open-sourced multi-modal data (see Figure 2) of
activities in-the-wild. The authors provide a baseline system for sensor
fusion and a unified model for multi-label classification. They trained
logistic regression and fully connected neural networks on hand-crafted
features that are extracted based on extensive domain-knowledge. In this
paper, we utilize this heterogeneous sensors data collected over a week from
sixty users to learn rich representations in an end-to-end fashion for
recognizing multi-label human behavioral context.

The task of learning detailed human context is challenging, especially from
imbalanced and multi-label data. Unconstrained device usage, a natural
environment, different routines, and authentic behaviors are likely to result
in a joint training dataset from several users with significant class skew (?)
and missing labels. Another challenge with learning from multi-modal signals
is developing an architecture that feasibly combines them as in diverse
environments a certain sensor might perform better than others. For instance,
if a person is watching a television with a phone lying on the table, the
sound modality may dominate in the network as compared to an accelerometer. We
address the former issue with instance weighting scheme same as (?) and later
through a unified architecture that can efficiently fuse representations in
multiple ways.

We present a deep temporal convolutional neural network (CNN) that learns
directly from various modalities through a multi-stream architecture
(accelerometer, gyroscope, sound and phone state networks). Here, a separate
network facilitates learning from each modality and a contextualization module
incorporates all the available information to determine the user’s context
(see Figure 1). In our experiments, we show that deep multi-modal
representations learned through our network without any sophisticated pre-
processing or manual feature extraction achieve state-of-the-art performance.

The primary contribution of this paper is in showing how to leverage ample
amount of raw sensory data to learn deep cross-modal representations for
multi-label behavioral context. Although, the methods in the paper are
standard, their application on a large-scale imbalanced and sparsely labeled
smartphone data set is unique. The proposed network architecture achieves
sensitivity and specificity score of $0.767$ and $0.733$, respectively
averaged over $51$ labels and $5$-folds cross-validation. The rest of the
paper describes our technique and experiments in detail. First, we review the
related work on activity recognition. Then we present our multi-stream
temporal convolutional network, architectural modifications for handling
missing sensors, the proposed training procedure and implementation details.
Next, the description of the dataset, evaluation protocol and experimental
results are described, followed by the conclusions.

Figure 3: End-to-end multi-modal and multi-label context recognition: We
propose a deep temporal convolutional architecture for multi-label behavioral
context recognition. A separate network learns representations (features) from
each modality using depthwise-separable convolutions and contextualizes this
information through shared layers to infer the user context.

## Related Work

Human activity recognition has been extensively studied in simulated and
controlled environments. It is concerned with classifying sensor measurements
into existing activity categories. The earlier techniques are predominantly
based on applying shallow learning algorithms on manually extracted features
(e.g. statistical and spectral attributes) (?). Despite there are unsupervised
(?; ?) and supervised (?; ?; ?; ?) deep learning methods applied for automatic
feature extraction to detect activities, these approaches are fairly limited
by the amount of labeled data (of many sensing modalities) from the real-
world. Furthermore, they do not fully address the issue of multi-label context
recognition. A user state is described by only one class or label, which is
not true for activities humans perform in real-life. Moreover, only recently
the exploration has begun into joint-learning and fusing multiple modalities
for ubiquitous sensing through deep networks (?; ?). The works cited here are
by no means an exhaustive list, but provide a recent representative
advancements made in utilizing deep neural networks for activity recognition.
We recommend the interested readers to refer (?; ?) for an extensive survey of
former approaches.

A systematic analysis of several deep neural architectures for activity
recognition is provided by ?. The suitability of various models is
investigated that were trained only on raw accelerometer signals for different
activity classification tasks. On diverse benchmark datasets, CNN and long-
short-term memory networks are found to outperform hand-crafted features by a
significant margin. Likewise, ? proposed an approach combining pre-training
and fine-tuning of deep belief networks for sequential activity recognition.
They extracted spectrograms from a triaxial accelerometer and found them to be
helpful for capturing variations in the input. Similarly, ? used $2$D activity
images extracted from accelerometer signals as CNN input. The importance of
unsupervised training of models in feature learning and optimization is
highlighted in (?) using a combination of sparse-coding framework and semi-
supervised learning. Likewise, ? developed a multi-channel CNN model to
replace heuristic based hand-crafted features. Their analysis showed CNNs work
well compared to traditional (shallow) learning algorithms on several
datasets. Audio sensing is also employed in unconstrained acoustic
environments through applying fully connected neural networks (?). Recently, ?
used deep networks for multi-modal activity recognition and compared them with
traditional learning algorithms on various recognition tasks. Likewise,
numerous other studies also positively utilize deep learning for detection of
basic activities (?; ?; ?).

We differentiate ourselves from the existing approaches through utilizing a
deep multi-stream CNN (with depthwise separable convolutions) on a large and
diverse context detection dataset. Specifically, we build on previous work by
? that only employed hand-engineered features for training linear and shallow
neural networks. In contrast, our general-purpose approach allows us to train
a deeper network that can not only automatically discover hidden latent
factors, but also seamlessly combine them to achieve an end-to-end learning
system without requiring domain expertise. Moreover, through taking advantage
of multi-task learning (?) we develop an architecture that can robustly handle
missing sensors.

## Learning Multi-Modal Networks

We design a deep convolutional neural network to address the problem of
behavioral context recognition through learning representations from raw
sensory inputs. To deal with cross-modality signals i.e. accelerometer (Acc),
gyroscope (Gyro), audio (MFCC/Aud), and phone state (PS), we use a multi-
stream architecture. The network comprises five main modules as demonstrated
in Figure 3. This section describes each component, presents a strategy to
modify the proposed architecture to handle missing sensors and provides the
implementation details.

### Modality Specific Networks

We present a deep multi-modal convolutional architecture for learning context
representations. We propose to use a series of depthwise-separable
convolutions (DPS-Conv) (?) for processing different components (or channels)
of raw signals. In general, CNNs are also found to be well suited for
processing $1$D sequences due to their ability to learn translation invariant
features, scale separation, and localization of filters across time and space
(?). DPS-Conv consists of two operations i.e. a depthwise convolution and a
pointwise (or $1$ x $1$) convolution. Specifically, the first function
(depthwise convolution) performs a convolution independently over each input
channel and it is followed by the second operation of $1$ x $1$ convolution
that projects the channels estimated by the earlier onto a distinct channel
space to have the same number of output filters (?). The intuition of this
formulation falls in line with the classical procedures utilized by domain
experts to extract several features from each signal component independently
(e.g. $x$, $y$ and $z$ constituents of an accelerometer) but pointwise
convolution goes one step further and tries to learn unified factors that may
capture relationships among independent elements. Moreover, separable
convolutions make efficient use of parameters as opposed to their classical
counterpart and this property has made them a very promising candidate for
contemporary architectures that run on smart devices with limited computing
and energy capabilities (?; ?). Formally, in case of $1$D input sequence
$\mathbf{x}$ of length $L$ with $M$ channels, the aforementioned operation can
be formulated as follows (?):

$\text{DepthwiseConv}(\mathbf{x},\mathbf{w})_{i}=\sum_{l}^{L}(\mathbf{x}[i:i+k-1]\odot\mathbf{w})_{l}$
$\text{PointwiseConv}(\mathbf{x},\mathbf{w})_{i}=\sum_{m}^{M}(\mathbf{x}[i:i+k-1]\cdot\mathbf{w})_{m}$
$\text{DepthwiseSeparableConv}(\mathbf{x},\mathbf{w_{d}},\mathbf{w_{p}})_{i}=\\\
\text{PointwiseConv}_{i}(\text{DepthwiseConv}_{i}(\mathbf{x}[i:i+k-1],\mathbf{w_{d}}),\mathbf{w_{p}}\\\
$

where $\odot$ is elements-wise product, $\mathbf{x}[i:j]$ represents a segment
of the complete sequence with adjacent columns from $i$ to $j$, and
$\mathbf{w}$ represents filter with receptive field size of $k$.

The proposed network takes four different signals as input, each with its
independent disjoint pathway in the earlier layers of the network. Towards the
end, they are merged into shared layers that are common across all modalities
that are described in the next subsection. This network configuration has the
benefit of not just extracting modality-specific (and channel-specific)
features but it can also feasibly extract mutual representations through
shared layers. Each of the presented Acc and Gyro networks consist of $2$
temporal convolution layers which act as feature extractors over raw signals
of dimensions $800$ x $3$. The convolution layers have kernel sizes of $64$
and $32$ with a stride of $2$ and each layer has $32$ and $64$ filters,
respectively. We use rectified linear activation in all the layers and apply
depth-wise L$2$-regularization with a rate of $0.0001$. The audio network
takes mel frequency cepstral coefficients (see Section Dataset and Modalities)
of size 420 x 13 as input and it has a similar architecture except the kernel
size, which is set to $8$ and $6$ in the first and second layers,
respectively. Likewise, the discrete attributes indicating PS are fed into a
single layer fully-connected (FC) network with $64$ units and L$1$-penalty is
used on the weights with a rate of $0.0001$. Furthermore, we explore different
mechanisms to get a fixed dimension vector from each modality that can be fed
into a shared network. Specifically, we use: a) global max pooling (GMP), b)
global average pooling (GAP), c) a FC layer, and d) exactly pass the
representations without any transformation to the shared network.

### Shared Network (Contextualization)

Given the concepts extracted from each modality, the shared network generates
a modal-agnostic representation. To achieve this, we fuse the output of
earlier networks either through concatenation or apply standard convolution
(only for Acc, Gyro and Aud). We then feed the output into $2$ FC layers
having $2048$, $1024$ hidden units, respectively. Same as earlier, we use
rectified linear non-linearity and L$1$-regularization with a weight decay
coefficient of $0.0001$. The final output layer contains $51$ units (one for
each label) with sigmoid activation. Figure 3 visualizes the sharing of the
network layers, where, earlier layers are modality specific but downstream
layers become more general.

### Missing Sensors

In a real-life setting, a context recognition system may encounter missing
modalities which can limit its inference capability. To make the model robust
against such a situation, we develop a multi-task network (?), where learning
from each sensor is posed as a task. The initial configuration of the model is
the same as before but an additional layer (of $128$ units for Acc, Gyro,
MFCC/Aud and $64$ units for PS) with a separate loss function is added after
only a single shared layer of $1024$ hidden units. Figure 4 provides a high-
level overview of the architecture. We employ joint-training (with a learning
rate of $0.0003$) on all the modalities through aggregating cost functions of
each model in order to get a total loss. This architectural configuration
allows not only to learn independent and shared factors but enables inference
even when any of the sensors is missing. It does so through averaging (which
can be weighted) over probabilities produced by the individual networks.

Figure 4: Handling Missing Sensors with a Multi-task Network: A variant of the
earlier defined architecture with additional task (modality-specific) layers
and a separate loss function for each modality. It is able to recognize user
context even if only one sensor is producing data and the others are
unavailable.

### Implementation and Training Details

The networks are implemented in Tensorflow (?) and the models are learned from
scratch; initializing the weights with Xavier technique (?). Dropout (?) is
applied on the hidden layers with a probability of $0.2$. We use the Adam
optimizer with a learning rate of $0.0001$ (unless mentioned otherwise) and
use a batch size of $100$. We optimize the model weights for a fixed number of
iterations (i.e. $15000$) with mini-batch stochastic gradient descent and
backpropagation using instance-weighted cross-entropy objective function:

$\mathcal{J}_{C}=\dfrac{1}{NC}\sum_{i=1}^{N}\sum_{c=1}^{C}\Psi_{i,c}\cdot\mathcal{L}_{CE}(\hat{y}_{i,c},y_{i,c})$
$\mathcal{L}_{ce}(\hat{y},y)=-[(y\log(\hat{y})+(1-y)\log(1-\hat{y}))]$

where $\mathcal{L}_{ce}$ is the binary cross-entropy loss, and $\Psi$ is an
instance-weighting matrix of size $N$ x $C$ (i.e. number of training examples
and total labels, respectively). The instance weights in $\Psi$ are assigned
by inverse class frequency. Likewise, the entries for the missing labels are
set to zero, to impose no contribution in the overall cost from such examples.

## Experimental Results

We conduct several experiments to analyze the capability of the proposed
method. First, we provide a brief description of the utilized dataset and
signals. Second, we describe the evaluation approach and metrics used to
determine the model’s performance on a multi-label and imbalanced dataset.
Finally, we discuss our empirical observations, effect of different
modalities’ representation, comparison of various procedures to learn shared
factors and visualization of the internal representation.

### Dataset and Modalities

We choose to learn discriminative representations directly from raw Acc, Gyro,
Aud/MFCC and PS attributes from a smartphone because of their wide
adoptability and ubiquity. For this purpose, we chose to leverage ExtraSensory
Dataset (?) since it is collected in a natural environment from users’
personal devices. The experimental setup was not scripted but data collection
was performed when participants were busy with their daily routines to capture
varied activities and context combinations, in-the-wild conditions. This data
source contains over $300,000$ multi-labeled instances (with classes such as
‘outside’, ‘at a restaurant’, ‘with friends’ from a total of $51$ labels) from
sixty users. The complete data collection protocol is described in (?). Here,
we provide a high-level overview of the signals that we used in this study.
The samples are collected for $20$ seconds duration every minute from tri-axis
Acc and Gyro at a sampling frequency of $40$Hz, mel frequency cepstral
coefficients (MFCCs) for $46$msec frame are extracted from Aud recorded at
$22,050$Hz. Likewise, several phone state binary features are also collected
such as those specifying, time of day, battery level, ringer mode and Wi-Fi
connection etc. A few randomly selected samples of these signals are
illustrated in Figure 2.

We seek to process raw sensory values without manual feature engineering.
Thus, the only pre-processing we applied is to transform variable length
inputs to an identical temporal length. For this purpose, the MFCCs of
environmental audio are repeated (along time dimension) to get equal size
input, this is reasonable for ambient soundscapes as we are not particularly
interested in inferring a specific sound event. Similarly, the Acc and Gyro
samples of varying sizes are zero-padded and instances, where MFCC length is
shorter than twenty are discarded. Furthermore, we treat Acc, Gyro and Aud as
$m$-channels inputs ($3$, $3$, and $13$ channels, respectively) as it allows
us to efficiently learn independent factors from every sensor axis, thus
maximally utilizing the large-scale dataset.

### Evaluation and Metrics

Our models are evaluated with five-folds cross-validation with the same
divisions of sixty users as of (?), where training and test folds contain $48$
and $12$ users, respectively. For hyper-parameter optimization, we use nested
cross-validation (?) by randomly dividing the training fold data into training
and validation sets with ratio of $80$-$20$. After hyper-parameters selection,
we train our models on the complete dataset of training folds (individually,
each time from scratch) and calculate metrics on the testing folds.
Furthermore, it is mentioned earlier that the considered dataset is highly
imbalanced with sparse labels. In this case, simply calculating naive accuracy
will be misleading due to not taking underrepresented classes into account.
Similarly, precision and f$1$-score are also very likely to be affected by the
class-skew due to involvement of true positives in the denominator. Hence, we
adopt a metric named balanced accuracy (BA) (?) as used in (?), which
incorporates both recall (or true positive rate) and true negative rate:
$\text{BA}=\frac{\text{Sensitivity}+\text{Specificity}}{2}$. BA can be
interpreted as average accuracy achieved on either class (positive or negative
regarding binary classification). It stays identical to traditional accuracy,
if a model performs equally well on each class but drops to a random chance
(i.e. $50$%) if a classifier performs poorly on a class with few instances
(?). We calculate BA for each label independently and average them afterwards
to get a trustworthy score of the model’s overall performance.

### Results and Analysis

#### Analysis of Fusing Multi-Modal Representations:

We quantify the effect of different procedures for getting a fixed dimension
feature vector from each modality-specific network and examine their fusion
through different configurations of the shared network. It is important to
note that, we keep an entire network’s configuration same but only the layers
under consideration are changed. Table 1 provides the averaged (metrics)
scores over $51$ contextual labels and $5$-folds as a result of applying
global (max and average) pooling, using FC layer or simply feeding the
extracted representations to the shared network for further processing. For
the latter, we explore learning mutual representation from Acc, Gyr, and
Aud/MFCC through an additional standard convolution layer and compare its
performance with directly using flattened representations. Our experiments
suggest that global max pooling (GMP) over each modality’s features
outperforms other utilized techniques; achieving BA of $0.750$ with a
sensitivity rate of $0.767$. We believe the reason is that, GMP is capable of
picking-up high-level shift-invariant features, which are most discriminative
among others. Figure 5 presents per label metrics for this network on all the
$51$ labels in the dataset. Specifically, we notice majority of the labels
have BA score in range of $70\%$-$80\%$.

Figure 5: Performance metrics per label of the best performing model (with GMP): The scores are averaged over $5$-folds cross-validation. Table 1: Multi-modal context recognition: The metrics are reported for $5$-folds cross-validation averaged over $51$ class labels. BA stands for balanced accuracy. | BA | Sensitivity | Specificity
---|---|---|---
GMP | 0.750 ($\pm$ 0.012) | 0.767 ($\pm$ 0.015) | 0.733 ($\pm$ 0.016)
GAP | 0.748 ($\pm$ 0.009) | 0.753 ($\pm$ 0.012) | 0.742 ($\pm$ 0.015)
FC | 0.744 ($\pm$ 0.009) | 0.735 ($\pm$ 0.014) | 0.753 ($\pm$ 0.008)
Flattened | 0.742 ($\pm$ 0.014) | 0.734 ($\pm$ 0.029) | 0.749 ($\pm$ 0.007)
Conv | 0.738 ($\pm$ 0.011) | 0.725 ($\pm$ 0.022) | 0.752 ($\pm$ 0.022)

#### Comparison of Convolution Variants:

We evaluate the complete multi-stream model through replacing only DPS-Conv
layers with standard convolution (Std-Conv) in modality-specific networks. We
did not observe major performance differences between the two models as shown
in Table 2. Nevertheless, a model with DPS-Conv should be preferred because of
having lower computational cost than Std-Conv (?).

Table 2: Performance evaluation with different convolution layers. | BA | Sensitivity | Specificity
---|---|---|---
Std-Conv | 0.751 ($\pm$ 0.011) | 0.750 ($\pm$ 0.017) | 0.751 ($\pm$ 0.007)
| DPS-Conv
---
0.750 ($\pm$ 0.012) | 0.767 ($\pm$ 0.015) | 0.733 ($\pm$ 0.016)

#### Quantifying Modality Influence:

To examine the effect of different combinations of sensors (or features
learned from them) on the recognition capability of the model, we experimented
with training several networks with modified architectures. Specifically, in
this case the model only consisted of layers that are relevant to the signals
under consideration e.g. for evaluating models with only Acc, Aud, and PS, we
removed the Gyro network entirely and then trained it end-to-end from scratch.
Table 3 shows the evaluation results that highlights the importance of joint-
learning and fusion of multiple modalities to improve detection rate.

Table 3: Effect of different modalities on recognition performance. | BA | Sensitivity | Specificity
---|---|---|---
Acc | 0.633 ($\pm$ 0.011) | 0.668 ($\pm$ 0.027) | 0.599 ($\pm$ 0.017)
Gyro | 0.639 ($\pm$ 0.011) | 0.638 ($\pm$ 0.017) | 0.640 ($\pm$ 0.020)
Aud | 0.669 ($\pm$ 0.024) | 0.731 ($\pm$ 0.028) | 0.608 ($\pm$ 0.025)
PS | 0.712 ($\pm$ 0.005) | 0.723 ($\pm$ 0.011) | 0.700 ($\pm$ 0.013)
Acc, Gyro, PS | 0.733 ($\pm$ 0.010) | 0.744 ($\pm$ 0.021) | 0.722 ($\pm$ 0.014)
Acc, Gyro, Aud | 0.708 ($\pm$ 0.010) | 0.722 ($\pm$ 0.027) | 0.693 ($\pm$ 0.012)
Acc, Aud, PS | 0.745 ($\pm$ 0.013) | 0.757 ($\pm$ 0.025) | 0.733 ($\pm$ 0.015)
Gyro, Aud, PS | 0.748 ($\pm$ 0.012) | 0.768 ($\pm$ 0.014) | 0.728 ($\pm$ 0.014)
All | 0.750 ($\pm$ 0.012) | 0.767 ($\pm$ 0.015) | 0.733 ($\pm$ 0.016)

#### Fusion and Effect of Missing Sensors:

We now evaluate the modified architecture’s predictive performance (presented
in Section Missing Sensors), confronting various combinations of missing
signals. Table 4 provides experimental results showing that the proposed
multi-task network can handle lost modalities, achieving similar BA score as
when separate models for each modality are developed (see Table 3). However,
this flexibility comes at the price of slightly lower BA but makes a model
capable of operation in the face of unavailable sensors.

Table 4: Assessment of multi-task network for handling missing modalities. Each row provide averaged metrics score as earlier but only mentioned modalities that are used for determining user’s context. | BA | SN | SP
---|---|---|---
Acc | 0.634 ($\pm$ 0.008) | 0.652 ($\pm$ 0.027) | 0.616 ($\pm$ 0.013)
Gyro | 0.619 ($\pm$ 0.016) | 0.632 ($\pm$ 0.040) | 0.606 ($\pm$ 0.023)
Aud | 0.656 ($\pm$ 0.026) | 0.670 ($\pm$ 0.046) | 0.641 ($\pm$ 0.015)
PS | 0.688 ($\pm$ 0.009) | 0.709 ($\pm$ 0.015) | 0.667 ($\pm$ 0.012)
Acc, Gyro | 0.646 ($\pm$ 0.009) | 0.670 ($\pm$ 0.028) | 0.622 ($\pm$ 0.018)
Acc, Aud | 0.687 ($\pm$ 0.015) | 0.695 ($\pm$ 0.035) | 0.679 ($\pm$ 0.008)
Acc, PS | 0.708 ($\pm$ 0.007) | 0.713 ($\pm$ 0.015) | 0.702 ($\pm$ 0.012)
Gyro, Aud | 0.687 ($\pm$ 0.020) | 0.699 ($\pm$ 0.045) | 0.676 ($\pm$ 0.015)
Gyro, PS | 0.708 ($\pm$ 0.007) | 0.719 ($\pm$ 0.023) | 0.696 ($\pm$ 0.019)
Aud, PS | 0.708 ($\pm$ 0.013) | 0.717 ($\pm$ 0.027) | 0.698 ($\pm$ 0.010)
Acc, Gyro, Aud | 0.690 ($\pm$ 0.012) | 0.703 ($\pm$ 0.031) | 0.677 ($\pm$ 0.011)
Acc, Gyro, PS | 0.705 ($\pm$ 0.007) | 0.714 ($\pm$ 0.023) | 0.696 ($\pm$ 0.019)
Acc, Aud, PS | 0.721 ($\pm$ 0.007) | 0.729 ($\pm$ 0.019) | 0.712 ($\pm$ 0.011)
Gyro, Aud, PS | 0.721 ($\pm$ 0.011) | 0.730 ($\pm$ 0.030) | 0.711 ($\pm$ 0.017)
All | 0.720 ($\pm$ 0.008) | 0.728 ($\pm$ 0.025) | 0.712 ($\pm$ 0.015)
Figure 6: Assessment of instance-weighting and regularization:  We determine
the impact of cost sensitive loss function and regularization (i.e. weight
decay and dropout) on the network’s predictive power. The results labeled
under standard are with both IW and regularization.

(a) Environment (b) Body State (c) Transportation Mode (d) Phone Position

Figure 7: t-SNE embeddings: We visualize the mutual features learned through
fusion of multiple modalities (from the last layer) in the shared network.
Four sets of mutually-exclusive labels are identified from multi-labeled data
to use during final visualization of semantically related clusters extracted
through t-SNE.

#### Reliance on Instance Weighting and Regularization:

Our results thus far have been obtained through training a model with cross-
entropy loss. This incorporated instance-weights to handle class-imbalance. To
test network’s dependence on the cost sensitive loss function
($\mathcal{J}_{c}$), we examined a model’s performance that is trained without
it. As expected, the overall BA score drastically drops to a random chance
(see Figure 6) with worse performance on positive samples in comparison with
the negative ones. Likewise, we also trained a model without any sort of
regularization i.e. removing dropout, L$1$ and L$2$ penalties from the
network. The average recall rate on the held-out testing folds dropped to
$0.58$ which can be an indication of overfitting the training set. Hence,
incorporating both instance-weighting (IW) and regularization improved
performance significantly in learning from this imbalanced dataset. However,
further work will be necessary to investigate other techniques for managing
(sparse) rare labels such as oversampling and data augmentation in case of
multi-labeled instances.

#### Visualization:

In order to illustrate the semantic relevance of the learned features, we
applied t-SNE (?) to project high-dimensional data to $2$D embedding. We take
the output of the last FC layer (see Figure 3) from the shared network by
feeding a limited (but randomly selected) subset of the dataset to extract the
embeddings. Further, as the data under consideration is multi-labeled, we
identified sets of mutually-exclusive labels (e.g. Indoors vs. Outside) that
can be used to color code the data points to visually identify meaningful
clusters. Figure 7 provides a visualization for various sets of labels
suggesting the network is able to disentangle possible factors of variation
that may distinguish a class from the rest in large-scale sensory data.
Furthermore, to get better insights in the diversity of the extracted features
from each modality, in Figure 8, we visualize the feature maps produced by the
first layer of the DPS-Conv layer of modal-specific networks.

Figure 8: Feature Maps from Modality-Specific Networks: Illustration of
randomly selected (learned) features from first layer of convolutional
networks. (a), (b) and (c) represent outputs from Acc, Gyro and Aud models,
respectively.

## Conclusions

In this work, we tackled the problem of multi-label behavioral context
recognition with deep multi-modal convolutional neural networks. We propose to
train an end-to-end model for jointly-learning from low-level sensory data
(accelerometer, gyroscope, audio and phone state) of smart devices collected
in-the-wild. Our empirical results demonstrated various strategies for
feasibly fusing representations learned from different modalities and
quantifying their contribution on the predictive performance. We also showed
that instance-weighted cross-entropy loss (as also leveraged in (?)) and
regularization schemes enable the model to generalize well on highly
imbalanced (sparsely labeled) dataset. Furthermore, we present a slight
modification in the proposed network’s architecture to handle missing sensors;
potentially taking advantage of multi-task learning. We believe, the proposed
methodology is generic enough and can be applied to other related problems of
learning from multivariate time series. Additionally, potential directions for
future work would involve developing techniques to handle imbalanced multi-
label data, optimal sensor selection to reduce computation and battery
consumption, and incorporating other analogous sensors to further improve the
detection rate.

Acknowledgment SCOTT (www.scott-project.eu) has received funding from the
Electronic Component Systems for European Leadership Joint Undertaking under
grant agreement No 737422. This Joint Undertaking receives support from the
European Union’s Horizon 2020 research and innovation programme and Austria,
Spain, Finland, Ireland, Sweden, Germany, Poland, Portugal, Netherlands,
Belgium, Norway.
Various icons used in the figures are created by Anuar Zhumaev, Tim Madle,
Korokoro, Gregor Cresnar, Shmidt Sergey, Hea Poh Lin, Natalia Jacquier, Trevor
Dsouza, Adrien Coquet, Alina Oleynik, Llisole, Alena, AdbA Icons, Jeevan
Kumar, Artdabana@Design, lipi, Alex Auda Samora, and Michelle Colonna from the
Noun Project.

## References

  * [Abadi et al. 2016] Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al.  2016\.  Tensorflow: A system for large-scale machine learning.  In OSDI, volume 16, 265–283.
  * [Alsheikh et al. 2016] Alsheikh, M. A.; Selim, A.; Niyato, D.; Doyle, L.; Lin, S.; and Tan, H.-P.  2016\.  Deep activity recognition models with triaxial accelerometers.  In AAAI Workshop: Artificial Intelligence Applied to Assistive Technologies and Smart Environments.
  * [Bai, Kolter, and Koltun 2018] Bai, S.; Kolter, J. Z.; and Koltun, V.  2018\.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  arXiv preprint arXiv:1803.01271.
  * [Bengio, Courville, and Vincent 2013] Bengio, Y.; Courville, A.; and Vincent, P.  2013\.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence 35(8):1798–1828.
  * [Bhattacharya et al. 2014] Bhattacharya, S.; Nurmi, P.; Hammerla, N.; and Plötz, T.  2014\.  Using unlabeled data in a sparse-coding framework for human activity recognition.  Pervasive and Mobile Computing 15:242–262.
  * [Brodersen et al. 2010] Brodersen, K. H.; Ong, C. S.; Stephan, K. E.; and Buhmann, J. M.  2010\.  The balanced accuracy and its posterior distribution.  In Pattern recognition (ICPR), 2010 20th international conference on, 3121–3124.  IEEE.
  * [Caruana 1997] Caruana, R.  1997\.  Multitask learning.  Machine learning 28(1):41–75.
  * [Cawley and Talbot 2010] Cawley, G. C., and Talbot, N. L.  2010\.  On over-fitting in model selection and subsequent selection bias in performance evaluation.  Journal of Machine Learning Research 11(Jul):2079–2107.
  * [Chollet 2017] Chollet, F.  2017\.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1251–1258.
  * [Figo et al. 2010] Figo, D.; Diniz, P. C.; Ferreira, D. R.; and Cardoso, J. M.  2010\.  Preprocessing techniques for context recognition from accelerometer data.  Personal and Ubiquitous Computing 14(7):645–662.
  * [Glorot and Bengio 2010] Glorot, X., and Bengio, Y.  2010\.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, 249–256.
  * [Hammerla, Halloran, and Ploetz 2016] Hammerla, N. Y.; Halloran, S.; and Ploetz, T.  2016\.  Deep, convolutional, and recurrent models for human activity recognition using wearables.  arXiv preprint arXiv:1604.08880.
  * [Hoseini-Tabatabaei, Gluhak, and Tafazolli 2013] Hoseini-Tabatabaei, S. A.; Gluhak, A.; and Tafazolli, R.  2013\.  A survey on smartphone-based systems for opportunistic user context recognition.  ACM Computing Surveys (CSUR) 45(3):27.
  * [Jiang and Yin 2015] Jiang, W., and Yin, Z.  2015\.  Human activity recognition using wearable sensors by deep convolutional neural networks.  In Proceedings of the 23rd ACM international conference on Multimedia, 1307–1310.  ACM.
  * [Kaiser, Gomez, and Chollet 2017] Kaiser, L.; Gomez, A. N.; and Chollet, F.  2017\.  Depthwise separable convolutions for neural machine translation.  arXiv preprint arXiv:1706.03059.
  * [Lane, Georgiev, and Qendro 2015] Lane, N. D.; Georgiev, P.; and Qendro, L.  2015\.  Deepear: robust smartphone audio sensing in unconstrained acoustic environments using deep learning.  In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 283–294.  ACM.
  * [Lara and Labrador 2013] Lara, O. D., and Labrador, M. A.  2013\.  A survey on human activity recognition using wearable sensors.  IEEE Communications Surveys and Tutorials 15(3):1192–1209.
  * [Lee et al. 2013] Lee, Y.; Min, C.; Hwang, C.; Lee, J.; Hwang, I.; Ju, Y.; Yoo, C.; Moon, M.; Lee, U.; and Song, J.  2013\.  Sociophone: Everyday face-to-face interaction monitoring platform using multi-phone sensor fusion.  In Proceeding of the 11th annual international conference on Mobile systems, applications, and services, 375–388.  ACM.
  * [Lin et al. 2012] Lin, M.; Lane, N. D.; Mohammod, M.; Yang, X.; Lu, H.; Cardone, G.; Ali, S.; Doryab, A.; Berke, E.; Campbell, A. T.; et al.  2012\.  Bewell+: multi-dimensional wellbeing monitoring with community-guided user feedback and energy optimization.  In Proceedings of the conference on Wireless Health, 10.  ACM.
  * [Lin et al. 2015] Lin, Q.; Zhang, D.; Connelly, K.; Ni, H.; Yu, Z.; and Zhou, X.  2015\.  Disorientation detection by mining gps trajectories for cognitively-impaired elders.  Pervasive and Mobile Computing 19:71–85.
  * [Lorincz et al. 2009] Lorincz, K.; Chen, B.-r.; Challen, G. W.; Chowdhury, A. R.; Patel, S.; Bonato, P.; Welsh, M.; et al.  2009\.  Mercury: a wearable sensor network platform for high-fidelity motion analysis.  In SenSys, volume 9, 183–196.
  * [Miluzzo et al. 2008] Miluzzo, E.; Lane, N. D.; Fodor, K.; Peterson, R.; Lu, H.; Musolesi, M.; Eisenman, S. B.; Zheng, X.; and Campbell, A. T.  2008\.  Sensing meets mobile social networks: the design, implementation and evaluation of the cenceme application.  In Proceedings of the 6th ACM conference on Embedded network sensor systems, 337–350.  ACM.
  * [Ordóñez and Roggen 2016] Ordóñez, F. J., and Roggen, D.  2016\.  Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.  Sensors 16(1):115.
  * [Plötz, Hammerla, and Olivier 2011] Plötz, T.; Hammerla, N. Y.; and Olivier, P.  2011\.  Feature learning for activity recognition in ubiquitous computing.  In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, volume 22, 1729.
  * [Rabbi et al. 2015] Rabbi, M.; Aung, M. H.; Zhang, M.; and Choudhury, T.  2015\.  Mybehavior: automatic personalized health feedback from user behaviors and preferences using smartphones.  In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 707–718.  ACM.
  * [Radu et al. 2018] Radu, V.; Tong, C.; Bhattacharya, S.; Lane, N. D.; Mascolo, C.; Marina, M. K.; and Kawsar, F.  2018\.  Multimodal deep learning for activity and context recognition.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(4):157.
  * [Rashidi and Cook 2009] Rashidi, P., and Cook, D. J.  2009\.  Keeping the resident in the loop: Adapting the smart home to the user.  IEEE Transactions on systems, man, and cybernetics-part A: systems and humans 39(5):949–959.
  * [Rashidi and Mihailidis 2013] Rashidi, P., and Mihailidis, A.  2013\.  A survey on ambient-assisted living tools for older adults.  IEEE journal of biomedical and health informatics 17(3):579–590.
  * [Ronao and Cho 2016] Ronao, C. A., and Cho, S.-B.  2016\.  Human activity recognition with smartphone sensors using deep learning neural networks.  Expert Systems with Applications 59:235–244.
  * [Sandler et al. 2018] Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C.  2018\.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4510–4520.
  * [Shoaib et al. 2015] Shoaib, M.; Bosch, S.; Incel, O. D.; Scholten, H.; and Havinga, P. J.  2015\.  A survey of online activity recognition using mobile phones.  Sensors 15(1):2059–2085.
  * [Srivastava et al. 2014] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R.  2014\.  Dropout: A simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research 15(1):1929–1958.
  * [Vaizman, Ellis, and Lanckriet 2017] Vaizman, Y.; Ellis, K.; and Lanckriet, G.  2017\.  Recognizing detailed human context in the wild from smartphones and smartwatches.  IEEE Pervasive Computing 16(4):62–74.
  * [Vaizman, Weibel, and Lanckriet 2018] Vaizman, Y.; Weibel, N.; and Lanckriet, G.  2018\.  Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(4):168.
  * [van der Maaten and Hinton 2008] van der Maaten, L., and Hinton, G.  2008\.  Visualizing data using t-sne.  Journal of machine learning research 9(Nov):2579–2605.
  * [Yang et al. 2015] Yang, J.; Nguyen, M. N.; San, P. P.; Li, X.; and Krishnaswamy, S.  2015\.  Deep convolutional neural networks on multichannel time series for human activity recognition.  In IJCAI, 3995–4001.
  * [Zeng et al. 2014] Zeng, M.; Nguyen, L. T.; Yu, B.; Mengshoel, O. J.; Zhu, J.; Wu, P.; and Zhang, J.  2014\.  Convolutional neural networks for human activity recognition using mobile sensors.  In Mobile Computing, Applications and Services (MobiCASE), 2014 6th International Conference on, 197–205.  IEEE.
  * [Zhang et al. 2017] Zhang, X.; Zhou, X.; Lin, M.; and Sun, J.  2017\.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  arXiv preprint arXiv:1707.01083.


# Design and characterisation of an antiproton deceleration beamline for the
PUMA experiment

J. Fischera Corresponding author<EMAIL_ADDRESS>A. Schmidta N.
Azaryanb F. Butinb J. Ferreira Somozab A. Hussona
C. Klinka A. Obertellia M. Schlaicha A. Sinturelb N. Thausb and F. Wienholtza

(aTechnische Universität Darmstadt, Institut für Kernphysik,
Schloßgartenstraße 9, 64289 Darmstadt, Germany
bCERN, 1211 Geneva 23, Switzerland)

##  1 Introduction

The spatial distribution of protons and neutrons at and beyond the nuclear
surface of atomic nuclei challenges nuclear theory. In particular, nuclei with
a neutron excess exhibit a so-called neutron skin, where the neutron density
distribution extends beyond the proton density distribution. The thickness of
the neutron skin is defined as the difference in root-mean-square radii of the
density distributions

$\displaystyle\Delta r_{\mathrm{np}}=\langle
r^{2}_{\mathrm{n}}\rangle^{1/2}-\langle r^{2}_{\mathrm{p}}\rangle^{1/2}.$ (1)

The neutron skin thickness correlates with the slope parameter $L$ of the
nuclear equation of state [1], playing an important role in defining the
relation between the mass and radius of a neutron star [2, 3]. Neutron skin
thicknesses have been investigated with several methods [4, 5, 6, 7, 8],
mostly on stable nuclei, while the challenge lies in determining the radius of
the neutron distribution $\langle r^{2}_{\mathrm{n}}\rangle^{1/2}$ with enough
accuracy and controlled theoretical uncertainties. Information on unstable
nuclei is much more scarce, as illustrated by Ca isotopes: charge radii can be
accessed with precision from the relative measurement of isotope shifts from
laser spectroscopy and anchored to stable nuclei [9], while the interpretation
of the data related to the matter or neutron radius suffers from model
dependence [10, 11]. Nuclei close or at the neutron drip line can have loosely
bound nucleons, whose wave function extends far beyond the charge
distribution. Such systems are called halo nuclei [12, 13]. Neutron halos have
been so far observed in light nuclei only [14]. Indications for $p$-wave halos
in medium mass nuclei have been reported [15], while more halos are predicted
to exist in uncharted regions of the nuclear landscape [16]. Proton halos have
been predicted as well [17].
Most aforementioned methods to probe neutron skins and halos in stable and
unstable nuclei are sensitive to the nuclear surface where
$\rho\sim\rho_{0}/2$, not further out in the tail of the density distribution,
where the asymmetry is the largest. The antiProton Unstable Matter
Annihilation (PUMA) experiment aims to investigate these phenomena in the tail
of stable and unstable nuclei with low-energy antiprotons as a probe [18, 19].
Antiprotons are uniquely suited for this, as they annihilate with nucleons at
a mean radial position $\sim 2$ $\mathrm{f}\mathrm{m}$ further out from the
half density radius of the nucleus [5, 20, 21], probing a region of higher
neutron-to-proton asymmetry. The PUMA experiment will produce antiprotonic
atoms by combining nuclei and antiprotons in a Penning trap. By studying the
pions produced in the annihilation, the PUMA experiment can determine the
neutron-to-proton ratio in the tail of the nuclear density distribution. The
setup is located at the Antimatter Factory at CERN. Stable isotopes are
supplied by an offline ion source [22], and for the investigation of more
neutron-rich and unstable isotopes the setup will be transported to the ISOLDE
facility [23] at CERN.
The ELENA ring at the Antimatter Factory provides bunches of $5\cdot 10^{6}$
to $10^{7}$ antiprotons at 100 $\mathrm{k}\mathrm{e}\mathrm{V}$ to up to four
experiments every 2 minutes [24, 25, 26]. To further decelerate the
antiprotons to energies compatible with the PUMA Penning trap, one can use a
thin degrader foil or pulsed drift tubes (PDT) [27, 28, 29, 30, 31, 32].
Employing a foil for deceleration is space-efficient, but the yield is low and
the energy distribution broad [33], compared to a pulsed drift tube, which can
have a transmission of 100% while conserving the width of the energy
distribution. For antiprotons with an initial energy of approximately 100 keV,
trapping efficiencies vary from a few percent [34] to a maximum of 50%,
predicted in [35]. However, for the PUMA experiment, which relies on the
simultaneous trapping of antiprotons and stable and unstable ions, the use of
a foil is unfeasible, since low-energy ions cannot penetrate the foil.
An established method to change the energy of a particle beam is to use a
drift tube, where the potential can be changed rapidly. Here, the drift tube
is set to a potential and is used to decelerate the particles to the desired
energy. If the electrode is switched to a different potential, e.g., ground,
while the particles are still inside and in the field free region of the drift
tube, they are not reaccelerated on exit. Because only the longitudinal and
not the transversal kinetic energy is changed, the divergence angle of the
beam increases by a factor of $\sqrt{E_{\mathrm{in}}/E_{\mathrm{out}}}$, where
$E$ is the kinetic energy of the incoming and outgoing particles,
respectively. This can be compensated by additional ion optical elements or
beam cooling.
Several ion trap experiments use pulsed drift tubes to decelerate nuclei for
trapping [22, 36, 37, 38], often in combination with buffer-gas cooling [39]
to counteract the increase in transversal emittance, some from energies as
high as $60$ $\mathrm{k}\mathrm{e}\mathrm{V}$. The GBAR experiment at CERN is
confronted with a similar problem as the PUMA experiment, as they need to
decelerate antiprotons to $1$ $\mathrm{k}\mathrm{e}\mathrm{V}$ [40]. At the
PUMA experiment, the antiprotons are decelerated from 100
$\mathrm{k}\mathrm{e}\mathrm{V}$ to 4 $\mathrm{k}\mathrm{e}\mathrm{V}$ to
allow for an efficient beam transport and in a second step down to 100
$\mathrm{e}\mathrm{V}$ right in front of the trap.
To limit the annihilation of antiprotons with residual gas molecules, a vacuum
of a few $10^{-10}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}$$ along and
$10^{-11}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}$$ at the end of the
beamline is critical.

##  2 Beamline Design

###  2.1 Transfer Line from ELENA to PUMA

Figure 1: Schematic view of LNE51 transfer line to PUMA. Antiprotons are
ejected from ELENA into LNE50, from which LNE51 branches off. The insert shows
the position of LNE51 relative to ELENA and LNE00.

The transfer of 100 keV particles (H- ions or antiprotons) from the ELENA
machine to the PUMA experiment is performed by the so-called LNE51 transfer
line. LNE51 branches off from the LNE50 line (transfer from ELENA to the
adjacent GBAR experiment) using a standard ZDFA-ZDSA switching unit (fast
switch and electrostatic deflector) integrated in LNE50. This equipment is
interlocked with the access safety system of the PUMA zone, preventing any
beam to be sent from the ELENA machine, while the area is being accessed. The
sector valve at the interface between the experiment and the LNE51 transfer
line is interlocked with the access system to close automatically when the
zone is being accessed. This drastically limits the risk of contamination of
the upstream sections of ELENA machine in case of an incident while
manipulating the experimental equipment. To satisfy the integration
constraints and match the beam to the PUMA experiment at the end of the line,
four electrostatic quadrupole/H-V corrector units (ZQNA) are installed, along
with a 37.7° standalone deflector. At the focal point, the beam spot size
(rms) is approximately 2 mm and the horizontal and vertical geometric
emittance ($95\%=6\epsilon_{\mathrm{rms}}$) is 6 mm mrad and 4 mm mrad,
respectively [41]. The layout for LNE51 is shown schematically in Figure 1.
Two SEM grids (Secondary Emission Monitors) [42] are installed in LNE51. They
are standard equipment in the ELENA transfer lines that allow to extract the
profile of the impinging beam, either H- ions or antiprotons. Made from x-y
meshes of 50 $\upmu$m tungsten wires, covering the beam acceptance, spaced by
a pitch of 0.5 mm in the central region, they intercept only about 10% of the
beam at each station [43]. These monitors are ultra-high vacuum compatible, as
they can be baked-out to 200°C. As bake-out is required, the vacuum line is
fitted with permanently installed bake-out jackets.

Figure 2: Half-section view of the beamline without the supports. The
antiprotons traverse the beamline from left to right. Ions from the offline
source enter the beamline at the quadrupole bender in the direction into the
page and are deflected to the right. The gate valves separating the sections
are depicted in blue.

###  2.2 The PUMA Antiproton Beamline

Downstream of the handover point (HOP) to PUMA (see Fig. 1 and 2), the
beamline consists of two main sections, that can be isolated by gate valves
type 48236-CE44 from VAT (see Fig. 2). Section 1 includes the pulsed drift
tube itself. It is complemented by a high-voltage (up to -85 kV) as well as a
low-voltage (up to 5 kV) einzel lens (EL) on the injection and ejection sides,
respectively, to focus the antiproton bunches into and out of the pulsed drift
tube.
Section 2 consists of two low-voltage (up to 5 kV) einzel lenses with x-y-
steerers to guide the beam to the entrance of the PUMA Penning trap. In
between these lenses, a quadrupole ion beam bender allows the injection of
ions from an offline ion source setup, perpendicular to the antiproton
beamline. Even tough the bender has been designed to allow for simultaneous
injection of ions and antiprotons, it can be removed when it is not needed. A
beam imaging system (BTV), which consists of a phosphorous screen and a
camera, completes the section. The BTV can be moved in and out of the
beamline, as it is a completely destructive measurement of the beam. In the
future, the BTV will be replaced by a SEM grid.

Figure 3: The field strength at an unshielded triple junction (left) and one
shielded with a guard ring (right) is illustrated here. Blue indicates lower
and red higher electric field strengths.

###  2.3 The Pulsed Drift Tube

The pulsed drift tube (PDT) used for the PUMA experiment, is based on the GBAR
design [30]. Although the high-voltage einzel lens in front of the drift tube
counteracts the strong focusing effect of the decelerating electric field, the
drift tube has to accommodate an expansion of the beam. The inner diameter was
thus chosen to be 100 $\mathrm{m}\mathrm{m}$ with an outer diameter of 120
$\mathrm{m}\mathrm{m}$. At 4 $\mathrm{k}\mathrm{e}\mathrm{V}$, an antiproton
bunch from ELENA has a length of 250 $\mathrm{m}\mathrm{m}$ ($2\sigma$) [26].
The PUMA pulsed drift tube has been designed to be 700 $\mathrm{m}\mathrm{m}$
long. This ensures, that the bunch is in the field free region of the drift
tube when the potential is changed.
Because of the stringent vacuum requirements
($p<10^{-10}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}$$), materials with the
lowest possible outgassing rates have to be used. Therefore, the pulsed drift
tube is made from aluminium ($\sim 1\cdot
10^{-13}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}~{}\mathrm{l}\mathrm{/}\mathrm{s}\mathrm{/}\mathrm{c}\mathrm{m}^{2}$$),
which outgases less than stainless steel ($\sim 3\cdot
10^{-12}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}~{}\mathrm{l}\mathrm{/}\mathrm{s}\mathrm{/}\mathrm{c}\mathrm{m}^{2}$$)
[44]. The insulators are made from MACOR®, which has an outgassing rate of
$1.1\cdot
10^{-11}\,$\mathrm{m}\mathrm{b}\mathrm{a}\mathrm{r}~{}\mathrm{l}\mathrm{/}\mathrm{s}\mathrm{/}\mathrm{c}\mathrm{m}^{2}$$
[45].
The walls of the vacuum chambers are coated with a non-evaporable getter (NEG)
to pump the section. Non-evaporable getters are made from an alloy of Zr, V,
Ti, Al and Fe, that can sputtered directly onto the wall of a vacuum chamber
[46]. It acts as a pump by absorbing hydrogen and chemically binding other
reactive gases like oxygen. To activate the NEG, the chambers are heated
(200°C to 400°C). Molecules at the surface (mainly carbon, nitrogen and
oxygen) diffuse into the bulk. Hydrogen is released and must be pumped away by
another pump. Therefore, all components, such as vacuum gauges, valves,
feedthroughs, pumps, cables and beam instrumentation, must be bakable at 250°C
at least. The coating of the inside surfaces of the chambers was done at CERN.
The installation of the pulsed drift tube inside the chamber must be done
without touching the coating to prevent damaging it. It is first mounted onto
its support structure before being lowered vertically into the vacuum chamber
and secured with screws. To facilitate individual access to the high- and low-
voltage einzel lens as well as the drift tube, the vacuum chamber is divided
into three parts.
At the intersections of vacuum, conductor and insulator, the electric field is
strongly enhanced due to gaps arising from imperfections on the corners of the
material (see Fig. 3). Special attention has been paid to these so-called
triple junctions to prevent possible discharges. They are shielded by purpose-
built rings, that surround the triple junction and thereby lower the electric
field (see Fig. 3). On all components, sharp edges have been avoided, and the
electrodes have been polished to an average surface finish of
$R_{\mathrm{a}}=0.05\,$\upmu\mathrm{m}$$, which helps to prevent discharges
[47].

####  2.3.1 Electronics

To not reaccelerate the antiprotons as they exit the pulsed drift tube, it
must be discharged from -96 kV to 0 V before the first antiprotons exit the
field free region of the drift tube. For antiprotons with a kinetic energy of
4 keV, the time to discharge the drift tube is in the order of 500 ns.
Equipment that can withstand high voltages and high peak currents, as well as
a high-voltage switch with a short transient, are needed. The pulsed drift
tube is connected to a high-voltage power supply (Spellman SL130PN60) via a
$1\text{\,}\mathrm{M\SIUnitSymbolOhm}$ resistor. In order not to exceed the
voltage rating of the resistors, two Metallux HVR 969 resistors are used,
connected via polished brass cylinders with rounded edges. The value is chosen
as a compromise between the need for a high resistance to decouple the power
supply from the pulsed drift tube while switching, and the need for a low
resistance to minimize the effects of current fluctuations on the voltage
applied to the pulsed drift tube. For the discharge of the tube’s capacitance,
a fast high-voltage switch (Behlke HTS 1501-20-LC2) connects the pulsed drift
tube to ground. To make sure that the switch is not damaged, the pulsed drift
tube is connected to the switch via a two
$250\text{\,}\mathrm{\SIUnitSymbolOhm}$ Metallux HVR 969 resistors in series,
limiting the current. The high-voltage leads are connected with HN-70
connectors from R.E. Beverly III & Associates. The cables are suspended from
the ceiling to avoid triple junctions at the exposed high-voltage connectors.
The grounded mesh is removed on the load side, and special care is taken to
cover the pointy ends of the grounded mesh. As high-voltage feedthrough, a
HV125R-CE-CU39 from VACOM, rated for up to 125 kV is used.

Using a 1/1000 voltage divider (LeCroy PPE6kV) connected to a Tektronix
MDO3104 oscilloscope, the switching time from -5 kV to ground was measured. As
can be seen in Fig. 4, there is a $\sim$250 ns delay between the trigger
signal (blue) and the voltage on the pulsed drift tube (orange) which has to
be taken into account when triggering the switch. Independent of the voltage
applied to the switch, the transient time $\tau$ to $V_{0}/\mathrm{e}$ is
$\sim$80 ns which is consistent with the time constant estimated by a simple
RC-circuit, where the capacitance of the pulsed drift tube was measured and
within the specs of the switch:

$\displaystyle\tau=RC=500\Upomega\cdot 170\mathrm{pF}=85\,\mathrm{ns}.$ (2)

Figure 4: Switching time while switching from $5\,$\mathrm{k}\mathrm{V}$$ to
ground, measured with a $1/1000$ voltage divider. The trigger signal is shown
in blue and the voltage on the pulsed drift tube in orange.

####  2.3.2 Safety Cage

The high-voltage system has unshielded $\sim$ 100 kV connections exposed to
air during operation. Therefore, the safety of the users has to be ensured by
a safety cage according to the ingress protection code level IP3X. Following
the European norm EN 50191, the dimensions of the safety cage are defined so
that any high-voltage point in air is at a distance of more than 74
$\mathrm{c}\mathrm{m}$ from the cage, corresponding to a maximum voltage of
130 kV, the maximum voltage of the high-voltage power supply. The high-voltage
system is interlocked via a switch (Telemecanique XCSDMC7902 coded magnetic
switch) at the sliding door of the cage to interlock the power supplies in the
event of unexpected access while the equipment is powered. The safety cage is
further secured with a trapped key system from Allen Bradley (Rockwell) to
prevent unauthorized access. It must first be locked to be able to switch on
the high-voltage power supplies. To simplify maintenance work, panels can be
removed from all sides of the cage.

##  3 Vacuum and Conditioning

###  3.1 Baseline Vacuum Pressure

Due to the strict vacuum requirements at the entrance of the PUMA trap,
special attention must be paid to the pressure. After activating the NEG
coating, a pressure of $2\cdot 10^{-11}$ mbar was measured at the end of the
pulsed drift tube section, a factor of 10 better than required. For the
subsequent tests, the NEG coating was not reactivated after venting, to
conserve it for the use with the PUMA trap attached. Without the NEG
activated, the pressure base level is around $1.4\cdot 10^{-10}$ mbar. This is
sufficient to condition and operate the pulsed drift tube.

###  3.2 High-Voltage Conditioning

Surface contamination and imperfections are sources of discharges that degrade
the vacuum and material when high voltage is applied. They also lead to a
leakage current that drains the set potential. This difficulty can be
countered by conditioning the high-voltage parts, which is therefore an
essential step before operating the pulsed drift tube. It was done by a
stepwise increase of the voltage, while keeping the leakage current below the
limit of the power supply and the vacuum better than $5\cdot 10^{-8}$ mbar.

The pulsed drift tube and high-voltage einzel lens were conditioned over
several weeks. The voltage was increased step by step and left in static
operation until the sudden spikes in current, associated with field emission
from imperfections on the electrode, subsided, which took between 12 and 72
hours per voltage step. In addition to the conditioning, modifications to the
setup were made outside the vacuum to reduce the leakage current. These
focussed on increasing the distance from any high-voltage parts to ground, as
well as polishing and rounding pieces in high electric fields. Ultimately, the
leakage current at -96 kV could be lowered from 100 $\upmu$A to 50 $\upmu$A by
polishing and increasing the corner radius of one high-voltage part from 3 mm
to 15 mm. Additionally, the current could be further decreased to 11 $\upmu$A
by increasing the ceiling height of the safety cage by 50 cm to 75 cm. The
leakage current of the high-voltage einzel lens could not be reduced in the
same way. At -85 kV, the 100 $\upmu$A current limit of the power supply is
reached. This means that the design value of -90 kV could not be achieved,
nevertheless it could be used for commissioning. A redesign with increased
distances between high-voltage parts and ground is planned.

###  3.3 Vacuum During Operation

During operation of the pulsed drift tube, the remaining leakage current
inside the vacuum degrades the pressure. To mitigate this, as done by the GBAR
collaboration, the voltage is kept at 0 V for most of the ELENA cycle and is
increased to -96 kV only 9.5 s before a bunch of antiprotons arrives. Ramping
up the voltage only shortly111compared to a repetition time of 120 s for
ELENA. before the bunch arrives has the advantage, that the vacuum is below
$2\cdot 10^{-10}$ mbar most of the time, since there is no leakage current at
0 V. When -96 kV are applied, the pressure reaches a value of $8\cdot
10^{-10}$ mbar and increases to $2\cdot 10^{-9}$ mbar when switching (see Fig.
5).

Figure 5: The pressure in section 1 and in section 2, while switching (three
cycles). This is without the NEG coating activated, to conserve it for the use
with the PUMA trap attached.

##  4 Measurement of Beam Properties

###  4.1 Detection System

For the characterization of the system, a vacuum chamber with several
detectors was installed at the end of the beamline. To visualize the beam
spot, a microchannel plate (MCP) by Hamamatsu with a phosphor screen with a
diameter of 40 mm was used. In combination with the camera CS505MU and lens
MVL7000 from Thorlabs, this results in the smallest resolvable feature being
40 $\upmu$m. The device was mounted on a tripod in front of a view port, which
allowed to capture the beam shape. A MagneToF detector by ETP ion detect was
used for two purposes: first, to determine the time of flight (ToF) of the
antiprotons (<1.5 ns multiple ion pulse width), and second, in combination
with an “energy grid”, to determine the kinetic energy distribution of the
decelerated antiprotons. The energy grid consists of a stack of three grids by
ETP ion detect with a diameter of 76.2 mm. The distance between the grids is
15 mm. The grid wires have a diameter of 0.018 mm, a centre-to-centre distance
of 0.25 mm, and a transmission of 92% to 95%. The two outer grids were
grounded, while a blocking voltage was applied to the middle one, with a
ripple of less than 10 mV. The energy grids and the MagneToF detector can be
moved out of the beam axis independently. In addition to those detectors, the
BTV further upstream in the beamline (see Fig. 2) was used for particle
detection and intensity determination.

###  4.2 Pulsed Drift Tube Switching Delay

When antiprotons arrive in the experimental zone, a trigger signal from the
ejection from ELENA is forwarded to the electronics. Relative to the trigger,
a switching time $t_{\mathrm{s}}$ has to be determined, at which the bunch is
fully contained inside the pulsed drift tube, so that the deceleration is
successful for the full antiproton bunch. To determine the ideal value,
$t_{\mathrm{s}}$ has to be scanned while observing the time of flight of the
antiprotons. If $t_{\mathrm{s}}$ is too small, the antiprotons see a grounded
electrode and traverse the pulsed drift tube at full speed, arriving the
earliest and with their initial energy. If $t_{\mathrm{s}}$ is too large, the
antiprotons are decelerated while entering the pulsed drift tube and
reaccelerated when leaving it, thus they arrive later than the ones never
decelerated, but still with their initial energy. When switching at the
correct time, the antiproton bunch is decelerated on entry but is not
reaccelerated on exit. Thus, it arrives later than in the other cases, as they
are slower, which can be seen in a simulation of the deceleration in the
pulsed drift tube performed in SIMION® (see top panel of Fig. 6).
The results from the measurement can be seen in the bottom panel of Fig. 6,
they match the behaviour expected from simulations. When $t_{\mathrm{s}}$ is
too small, the antiprotons arrived early. When increasing $t_{\mathrm{s}}$,
the bunch diffuses, as the bunch is partly in the fringe field of the
electrode when the pulsed drift tube is switched. Afterwards, in a window of
about 300 ns, the antiprotons are uniformly decelerated. As $t_{\mathrm{s}}$
is further increasing, the bunch diffuses again, because it is only partly
inside the pulsed drift tube when switching.

Figure 6: Simulated (top) and measured (bottom) beam intensity when switching
the pulsed drift tube from $-96\,$\mathrm{k}\mathrm{V}$$ to ground and varying
the switch delay $t_{\mathrm{s}}$. Yellow colours indicate lower and red
higher intensity. In both cases, a successful deceleration to
$4\,$\mathrm{k}\mathrm{e}\mathrm{V}$$ corresponds to a time of flight of
$t_{\mathrm{4keV}}=3.85\,$\upmu\mathrm{s}$$, with a bunch length ($1\sigma$)
of $0.09\,$\upmu\mathrm{s}$$. On the right, the integrated intensity from
$t_{\mathrm{4keV}}-2\sigma$ to $t_{\mathrm{4keV}}+2\sigma$ is shown,
$t_{\mathrm{s}}$ is chosen to maximise this intensity.

The measurement shows a successful deceleration, and an estimation with the
time of flight gives a deceleration to $(4.0\pm
0.5)\,$\mathrm{k}\mathrm{e}\mathrm{V}$$. A more precise measurement of the
energy distribution was done using the energy grids (see Sec. 4.4).

###  4.3 Transmission and Focusing

The intensity of the bunch after the pulsed drift tube $I$, can be compared to
the initial intensity of the bunch $I_{0}$. The total transmission through the
pulsed drift tube is thus defined by $T=I/I_{0}$. $I_{0}$ is determined before
the handover point by pick-ups in the ELENA transfer lines [24]. Besides
showing the beam spot shape, the total intensity on the BTV is proportional to
$I$, as can be seen in Fig. 7. Using the calibration in this plot, $T$ can be
calculated.

Figure 7: The bunch intensity of antiprotons determined by the ELENA detectors
is proportional to total intensity on the BTV. The transmission to the BTV is
$100\%$ when not decelerating the antiprotons. This allows to make a
calibration to determine the transmission through the pulsed drift tube while
decelerating.

$T$ for 100 keV bunches is about 100%. During the experiment, the transmission
of antiprotons decelerated to 4 keV reached ($55\pm 3$)%, while in simulations
a transmission of 100% could be reached. The main source of losses in
transmission can be assigned to a misalignment of the high-voltage einzel lens
and the pulsed drift tube and a high leakage current on the high-voltage
einzel lens, which limited the voltage to -85 kV. In addition, the parameters
assumed in the simulation for the incoming beam might also play a role. Figure
8 shows the beam profiles recorded by the BTV directly after the last einzel
lens. Using a Gaussian fit, the following parameters could be obtained:

$\displaystyle\sigma_{\mathrm{horiz}}=(3.0\pm
0.1)\,$\mathrm{m}\mathrm{m}$,\,\sigma_{\mathrm{vert}}=(3.8\pm
0.2)\,$\mathrm{m}\mathrm{m}$$

64% of the antiprotons are within a circle of radius
$r=5.6\,$\mathrm{m}\mathrm{m}$$, the smallest aperture of the PUMA Penning
trap. The focal point will have to be optimized at a later point for the
injection into the PUMA trap.

Figure 8: Beam profile after optimizing the LV einzel lenses for deceleration
to $4\text{\,}\mathrm{keV}$ and focus on the BTV. Fitting a Gaussian to the
centre peak yields
$\sigma_{\mathrm{horiz}}=3.0\,$\mathrm{m}\mathrm{m}$,\sigma_{\mathrm{vert}}=3.8\,$\mathrm{m}\mathrm{m}$$.
Yellow indicates a lower and red a higher intensity.

###  4.4 Energy Distribution

The standard deviation of the ions’ energy after deceleration to 4 keV at the
position of the MagneToF detector was simulated to be 101 eV. The kinetic
energy $E$ of the antiprotons was determined by blocking the antiprotons with
the energy grids, and measuring the transmission on the MagneToF. The results
from this can be seen in Fig. 9. In blue, the transmission onto the MagneToF
is displayed in dependence of the kinetic energy of the antiprotons. Fitting
the cumulative distribution function (CDF) of a normal distribution yields the
mean energy $\mu=(3898\pm 3)$ eV and energy spread $\sigma=(127\pm 4)$ eV. The
energy distribution calculated from the fit is shown in orange. 88% of
decelerated antiprotons are within $\pm$ 200 eV of the central energy, which
is the energy acceptance for successful trapping in the PUMA Penning trap,
according to simulations.

Figure 9: The energy distribution of decelerated antiprotons. The data and
fitted CDF of a normal distribution are shown in blue, and the probability
density function corresponding to the fit in orange. The mean energy is
$\mu=(3898\pm 3)$ $\mathrm{e}\mathrm{V}$ and the standard deviation
$\sigma=127\pm 4$ $\mathrm{e}\mathrm{V}$. $88\%$ of decelerated antiprotons
are within $\pm 200\,$\mathrm{e}\mathrm{V}$$ of the mean energy, which is the
estimated energy acceptance for trapping.

###  4.5 Bunch Length

The length of the antiproton bunch at 4 keV is relevant, because it determines
the losses in the second stage of deceleration to a few 100 eV right in front
of the trap. The simulation predicts an increase in length from 75 ns to 89 ns
at the position of the MagnetToF, with which 90% of the bunch can be trapped.
A measurement of the bunch length of the decelerated antiprotons with the
MagneToF yields a length ($1\sigma$) of 93$\pm$3 ns, consistent with the
simulation.

##  5 Conclusion

An overview of the design and the characterisation of the low-energy
antiproton beam line of PUMA at ELENA is presented. Design considerations for
high voltage and ultra-high vacuum are discussed, as well as procedures for
high-voltage conditioning and in-vacuum high-voltage operation. The antiproton
beamline is shown to be successful in decelerating antiprotons from 100 keV to
$(3898\pm 3)$ eV, the first step in trapping antiprotons for the PUMA
experiment. The pressure, with the pulsed drift tube not in operation, is
below $2\cdot 10^{-10}$ mbar. With the implemented high-voltage ramping
scheme, the pressure stays below $2\cdot 10^{-10}$ mbar 75% of the cycle time,
also during operation. Currently, a transmission of ($55\pm 3$)% for
antiprotons decelerated to 4 keV can be reached. The beam was focussed to a
spot with $\sigma_{\mathrm{horiz}}=(3.0\pm 0.1)\,$\mathrm{m}\mathrm{m}$,$ and
$\sigma_{\mathrm{vert}}=(3.8\pm 0.2)\,$\mathrm{m}\mathrm{m}$$, demonstrating
it can be focussed into the PUMA Penning trap. The length of the 4 keV
antiproton bunch, relevant for the second deceleration from 4 keV to 100 eV is
(93$\pm$3) ns. Further improvement of the beamline is foreseen in the future,
while the current performance already allows for first experiments with PUMA.

## Acknowledgements

We thank the ELENA team and the operators of the Antimatter Factory for
excellent beam during the runs in 2022 and 2023. We thank the technical teams
at CERN and TU Darmstadt for their support. The presented work benefited from
the support of the GBAR collaboration. PUMA is funded by the European Research
Council through the ERC grant PUMA-726276 and the Alexander-von-Humboldt
foundation. The development of PUMA and its implementation at CERN are
supported by the TU Darmstadt and CERN.

## References

  * [1] X. Roca-Maza, M. Centelles, X. Viñas and M. Warda  “Neutron Skin of 208Pb, Nuclear Symmetry Energy, and the Parity Radius Experiment”  In _Phys. Rev. Lett._ 106 American Physical Society, 2011, pp. 252501  DOI: 10.1103/PhysRevLett.106.252501
  * [2] K. Hebeler, J.. Lattimer, C.. Pethick and A. Schwenk  “Equation of State and Neutron Star Properties Constrained by Nuclear Physics and Observation”  In _Astrophys. J._ 773.1 The American Astronomical Society, 2013, pp. 11  DOI: 10.1088/0004-637X/773/1/11
  * [3] C.. Horowitz and J. Piekarewicz  “Neutron Star Structure and the Neutron Radius of ${}^{208}\mathrm{Pb}$”  In _Phys. Rev. Lett._ 86 American Physical Society, 2001, pp. 5647–5650  DOI: 10.1103/PhysRevLett.86.5647
  * [4] A. Krasznahorkay et al.  “Neutron-skin thickness in neutron-rich isotopes”  In _Nucl. Phys. A_ 731, 2004, pp. 224–234  DOI: https://doi.org/10.1016/j.nuclphysa.2003.11.034
  * [5] B. Kłos et al.  “Neutron density distributions from antiprotonic ${}^{208}\mathrm{Pb}$ and ${}^{209}\mathrm{Bi}$ atoms”  In _Phys. Rev. C_ 76 American Physical Society, 2007, pp. 014311  DOI: 10.1103/PhysRevC.76.014311
  * [6] J. Zenihiro et al.  “Neutron density distributions of ${}^{204,206,208}\mathrm{Pb}$ deduced via proton elastic scattering at ${E}_{p}=295$ MeV”  In _Phys. Rev. C_ 82 American Physical Society, 2010, pp. 044611  DOI: 10.1103/PhysRevC.82.044611
  * [7] C.. Tarbert et al.  “Neutron Skin of ${}^{208}\mathrm{Pb}$ from Coherent Pion Photoproduction”  In _Phys. Rev. Lett._ 112 American Physical Society, 2014, pp. 242502  DOI: 10.1103/PhysRevLett.112.242502
  * [8] D. Adhikari et al.  “Accurate Determination of the Neutron Skin Thickness of ${}^{208}\mathrm{Pb}$ through Parity-Violation in Electron Scattering”  In _Phys. Rev. Lett._ 126 American Physical Society, 2021, pp. 172502  DOI: 10.1103/PhysRevLett.126.172502
  * [9] R.. Garcia Ruiz et al.  “Unexpectedly large charge radii of neutron-rich calcium isotopes”  In _Nat. Phys._ 12.6, 2016, pp. 594–598  DOI: 10.1038/nphys3645
  * [10] Tomotsugu Wakasa et al.  “Neutron-skin values and matter and neutron radii determined from reaction cross sections of proton scattering on ${}^{12}\mathrm{C}$, ${}^{40,48}\mathrm{Ca}$, ${}^{58}\mathrm{Ni}$, and ${}^{208}\mathrm{Pb}$”  In _Phys. Rev. C_ 107 American Physical Society, 2023, pp. 024608  DOI: 10.1103/PhysRevC.107.024608
  * [11] M. Enciu et al.  “Extended ${p}_{3/2}$ Neutron Orbital and the $N=32$ Shell Closure in ${}^{52}\mathrm{Ca}$”  In _Phys. Rev. Lett._ 129 American Physical Society, 2022, pp. 262501  DOI: 10.1103/PhysRevLett.129.262501
  * [12] I. Tanihata et al.  “Measurements of Interaction Cross Sections and Nuclear Radii in the Light $p$-Shell Region”  In _Phys. Rev. Lett._ 55 American Physical Society, 1985, pp. 2676–2679  DOI: 10.1103/PhysRevLett.55.2676
  * [13] P G Hansen, A S Jensen and B Jonson  “Nuclear Halos”  In _Annu. Rev. Nucl. Part S_ 45.1, 1995, pp. 591–634  DOI: 10.1146/annurev.ns.45.120195.003111
  * [14] Björn Jonson  “Light dripline nuclei”  In _Phys. Rep._ 389.1, 2004, pp. 1–59  DOI: https://doi.org/10.1016/j.physrep.2003.07.004
  * [15] Manju, Jagjit Singh, Shubhchintak and R. Chatterjee  “Low-lying dipole strengths for probable p-wave one-neutron halos in the medium mass region”  In _Eur. Phys. J. A_ 55.1, 2019, pp. 5  DOI: 10.1140/epja/i2019-12679-4
  * [16] V. Rotival, K. Bennaceur and T. Duguet  “Halo phenomenon in finite many-fermion systems: Atom-positron complexes and large-scale study of atomic nuclei”  In _Phys. Rev. C_ 79 American Physical Society, 2009, pp. 054309  DOI: 10.1103/PhysRevC.79.054309
  * [17] Zhongzhou Ren, Baoqiu Chen, Zhongyu Ma and Gongou Xu  “One-proton halo in ${}^{26}\mathrm{P}$ and two-proton halo in ${}^{27}\mathrm{S}$”  In _Phys. Rev. C_ 53 American Physical Society, 1996, pp. R572–R575  DOI: 10.1103/PhysRevC.53.R572
  * [18] Michiharu Wada and Yasunori Yamazaki  “Technical developments toward antiprotonic atoms for nuclear structure studies of radioactive nuclei” Low Energy Antiproton Physics (LEAP’03)  In _Nucl. Instrum. Methods Phys. Res., Sect. B_ 214, 2004, pp. 196–200  DOI: https://doi.org/10.1016/j.nimb.2003.08.019
  * [19] T. Aumann et al.  “PUMA, antiProton unstable matter annihilation - PUMA collaboration”  In _Eur. Phys. J. A_ 58.5, 2022, pp. 88  DOI: 10.1140/epja/s10050-022-00713-x
  * [20] M. Leon and R. Seki  “Determination of the neutron halo from antiproton absorption”  In _Phys. Lett. B_ 48.3, 1974, pp. 173–175  DOI: https://doi.org/10.1016/0370-2693(74)90001-X
  * [21] A.S. Iljinov, V.I. Nazaruk and S.E. Chigrinov  “Nuclear absorption of stopped antiprotons: Multipion-nucleus interactions”  In _Nucl. Phys. A_ 382.3, 1982, pp. 378–400  DOI: https://doi.org/10.1016/0375-9474(82)90352-9
  * [22] M. Schlaich et al.  “A multi-reflection time-of-flight mass spectrometer for the offline ion source of the PUMA experiment”  In _Int. J. Mass Spectrom._ 495, 2024, pp. 117166  DOI: https://doi.org/10.1016/j.ijms.2023.117166
  * [23] María J G Borge and Klaus Blaum  “Focus on Exotic Beams at ISOLDE: A Laboratory Portrait”  In _J. Phys. G: Nucl. Part. Phys._ 45.1 IOP Publishing, 2017, pp. 010301  DOI: 10.1088/1361-6471/aa990f
  * [24] Wolfgang Bartmann et al.  “The ELENA facility”  In _Philos. Trans. R. Soc., A_ 376.2116, 2018, pp. 20170266  DOI: 10.1098/rsta.2017.0266
  * [25] Stephan Maury et al.  “ELENA: the extra low energy anti-proton facility at CERN”  In _Hyperfine Interact._ 229.1, 2014, pp. 105–115  DOI: https://doi.org/10.1016/0370-2693(74)90001-X
  * [26] Laurette Ponce et al.  “ELENA - From Commissioning to Operation”  In _JACoW IPAC_ 2022, 2022, pp. 2391–2394  DOI: 10.18429/JACoW-IPAC2022-THOXGD1
  * [27] H. Kalinowsky  “Deceleration of antiprotons from MeV to keV energies”  In _Hyperfine Interact._ 76.1, 1993, pp. 73–80  DOI: 10.1007/BF02316707
  * [28] C. Amole et al.  “The ALPHA antihydrogen trapping apparatus”  In _Nucl. Instrum. Methods Phys. Res., Sect. A_ 735, 2014, pp. 319–340  DOI: https://doi.org/10.1016/j.nima.2013.09.043
  * [29] M. Tajima et al.  “Antiproton beams with low energy spread for antihydrogen production”  In _J. Instrum._ 14.05, 2019, pp. P05009  DOI: 10.1088/1748-0221/14/05/P05009
  * [30] A. Husson et al.  “A pulsed high-voltage decelerator system to deliver low-energy antiprotons”  In _Nucl. Instrum. Methods Phys. Res., Sect. A_ 1002, 2021, pp. 165245  DOI: https://doi.org/10.1016/j.nima.2021.165245
  * [31] Claude Amsler et al.  “Pulsed production of antihydrogen”  In _Commun. Phys._ 4.1, 2021, pp. 19  DOI: 10.1038/s42005-020-00494-z
  * [32] B.. Latacz et al.  “Ultra-thin polymer foil cryogenic window for antiproton deceleration and storage”  In _Rev. Sci. Instrum._ 94.10, 2023, pp. 103310  DOI: 10.1063/5.0167262
  * [33] K. Nordlund, M. Hori and D. Sundholm  “Large nuclear scattering effects in antiproton transmission through polymer and metal-coated foils”  In _Phys. Rev. A_ 106 American Physical Society, 2022, pp. 012803  DOI: 10.1103/PhysRevA.106.012803
  * [34] N. Kuroda et al.  “Confinement of a Large Number of Antiprotons and Production of an Ultraslow Antiproton Beam”  In _Phys. Rev. Lett._ 94 American Physical Society, 2005, pp. 023401  DOI: 10.1103/PhysRevLett.94.023401
  * [35] S.S. Fabbri and W. Bertsche  “Optimization of Antiproton Capture for Antihydrogen Creation in the ALPHA Experiment” https://doi.org/10.18429/JACoW-IBIC2019-WEPP040  In _Proc. IBIC’19_ , International Beam Instrumentation Conferenc 8  JACoW Publishing, Geneva, Switzerland, 2019, pp. 637–641  DOI: 10.18429/JACoW-IBIC2019-WEPP040
  * [36] F Herfurth et al.  “A linear radiofrequency ion trap for accumulation, bunching, and emittance improvement of radioactive ion beams”  In _Nucl. Instrum. Methods Phys. Res., Sect. A_ 469.2, 2001, pp. 254–275  DOI: https://doi.org/10.1016/S0168-9002(01)00168-1
  * [37] S. Coeck et al.  “A pulsed drift cavity to capture 30keV ion bunches at ground potential”  In _Nucl. Instrum. Methods Phys. Res., Sect. A_ 572.2, 2007, pp. 585–595  DOI: https://doi.org/10.1016/j.nima.2006.11.054
  * [38] J. Grund et al.  “First online operation of TRIGA-TRAP”  In _Nucl. Instrum. Methods Phys. Res., Sect. A_ 972, 2020, pp. 164013  DOI: https://doi.org/10.1016/j.nima.2020.164013
  * [39] G. Savard et al.  “A new cooling technique for heavy ions in a Penning trap”  In _Phys. Lett. A_ 158.5, 1991, pp. 247–252  DOI: https://doi.org/10.1016/0375-9601(91)91008-2
  * [40] P. Pérez et al.  “The GBAR antimatter gravity experiment”  In _Hyperfine Interact._ 233.1, 2015, pp. 21–27  DOI: 10.1007/s10751-015-1154-8
  * [41] M.A. Fraser, D. Barna, W. Bartmann and R. Ostojić  “Beam Dynamics Studies of the ELENA Electrostatic Transfer Lines” https://doi.org/10.18429/JACoW-IPAC2015-MOPJE044  In _Proc. IPAC’15, Richmond, VA, USA, May 3-8, 2015_ , International Particle Accelerator Conference 6  Geneva, Switzerland: JACoW, 2015, pp. 385–388  DOI: https://doi.org/10.18429/JACoW-IPAC2015-MOPJE044
  * [42] M Martini and H Schönauer  “Emittance measurements in the CERN PS complex”, 1997
  * [43] M. McLean, J. Cenede, M. Hori and G. Tranquille  “Commissioning of the SEM-Grid Monitors for ELENA” https://doi.org/10.18429/JACoW-IBIC2021-TUPP14  In _Proc. IBIC’21_ , International Beam Instrumentation Conference 10  JACoW Publishing, Geneva, Switzerland, 2021, pp. 223–226  DOI: 10.18429/JACoW-IBIC2021-TUPP14
  * [44] C Benvenuti  “Extreme Vacua: Achievements and Expectations”  In _Phys. Scr._ 1988.T22, 1988, pp. 48  DOI: 10.1088/0031-8949/1988/T22/006
  * [45] Katharina Battes, Christian Day and Volker Hauer  “Systematic study of the outgassing behavior of different ceramic materials”  In _J. Vac. Sci. Technol., B_ 39.3, 2021, pp. 034202  DOI: 10.1116/6.0000954
  * [46] P. Chiggiato and P. Costa Pinto  “Ti–Zr–V non-evaporable getter films: From development to large scale production for the Large Hadron Collider” Proceedings of the Eighth International Conference on Atomically Controlled Surfaces, Interfaces and Nanostructures and the Thirteenth International Congress on Thin Films  In _Thin Solid Films_ 515.2, 2006, pp. 382–388  DOI: https://doi.org/10.1016/j.tsf.2005.12.218
  * [47] D W Williams and W T Williams  “Effect of electrode surface finish on electrical breakdown in vacuum”  In _J. Phys. D: Appl. Phys._ 5.10, 1972, pp. 1845  DOI: 10.1088/0022-3727/5/10/314