# A mixed-reality dataset for category-level 6D pose and size estimation of
hand-occluded containers

###### Abstract

Estimating the 6D pose and size of household containers is challenging due to
large intra-class variations in the object properties, such as shape, size,
appearance, and transparency. The task is made more difficult when these
objects are held and manipulated by a person due to varying degrees of hand
occlusions caused by the type of grasps and by the viewpoint of the camera
observing the person holding the object. In this paper, we present a mixed-
reality dataset of hand-occluded containers for category-level 6D object pose
and size estimation. The dataset consists of 138,240 images of rendered hands
and forearms holding 48 synthetic objects, split into 3 grasp categories over
30 real backgrounds. We re-train and test an existing model for 6D object pose
estimation on our mixed-reality dataset. We discuss the impact of the use of
this dataset in improving the task of 6D pose and size estimation.

##  1 Introduction

Estimating the position and orientation in 3D (or 6D pose) of an object from a
single view is important for augmented reality, immersive online gaming,
online shopping, robotic grasping and manipulation, or human-to-robot
handovers in assistive scenarios [1, 2, 3, 4]. Despite being investigated for
a long time, the problem is still challenging and far from solved due to the
large variety of objects and their physical properties, the differences in
environmental conditions, and the uncertainty in the applicative scenarios.
For example, objects vary in their shape, size, appearance (different textures
or lack of texture, colour), and transparency [5, 6, 7]. These physical
properties can also vary within instances of a same category, such as food
boxes, containers for liquids, cameras, or shoes [4, 8]. Moreover, some
objects can be manipulated by people (e.g., in the kitchen [9]), resulting in
increased and more uncertain variations in the object poses and different
degrees of hand occlusions.

Early approaches rely on the availability of 3D models of opaque objects with
enough texture to identify corner-like points in the image and estimate the
pose with a close-form solution or an optimisation strategy [10]. Approaches
based on deep neural networks use large and diverse amounts of data to train a
model to predict the pose of objects with different properties [5, 6, 11]. The
task is nowadays often combined with segmentation of the object(s) in the
image, or with other tasks, such as shape reconstruction [4, 6, 12] (see Fig.
1).

Fig. 1: 6D pose estimation for a container manipulated by a person on our
mixed-reality (left) and on CORSMAL Containers Manipulation [13] (right)
datasets. A model [4] can first identify and localise the container on the
image, predict the 3D normalised coordinates, and then recover the pose and
size of the object in 3D.

Because of the dependency on data to train the models, there has been an
increase of publicly available datasets for object pose estimation. Each
dataset focuses on a specific property of a set of objects or object
categories (e.g., lack of texture, shape variations, transparency), or on a
specific purpose (e.g., grasping objects lying on a surface, reconstruction of
objects and hands during a human manipulation of the object). Examples of
these datasets are LineMOD [14], HomebrewedDB [15], YCB-Video [7], TUD-L [5]
TOD [11], HO3D [16], NOCS [4], Objectron [8], and OBMan [17]. TOD and YCB-
Video focus on specific object instances and therefore cannot be used to
generalise to large variability between objects of the same class. Objectron
and NOCS consist of image sequences, captured by a moving camera, of single or
multiple objects lying on a surface for estimating the pose and size of these
objects belonging to a set of categories. Unlike a cut-and-paste approach [18]
that cannot provide annotations of the 6D object pose, NOCS also generates
images of synthetic objects rendered on real background scenes in a context-
manner (mixed-reality) to handle the necessity of large data with
corresponding free annotations of the 3D normalised coordinated maps. However,
all the above-mentioned datasets do not include manipulation of the objects by
a person, thus limiting the variability in the object poses to only those
related to the camera motion. HO3D and OBMan address handheld objects for
reconstructing or estimating the pose of both the human hand and the
manipulated object. However, HO3D considers a limited subset of object
instances from the list of YCB objects, therefore lacking any intra-class
variability. OBMan is a category-level based and object-centric dataset that
randomly renders synthetic objects and hands on top of real backgrounds.
Datasets such as OBMan and the mixed-reality part of NOCS avoid the collection
of real data and their annotation that may be cumbersome, span a large amount
of time, and require the involvement of many people as well as the use of
additional (expensive) equipment, while allowing the generation of large
composite image data accompanied by free annotations. However, the random-
based data augmentation approach to generate OBMan lacks a context-based
composition with the scene and plausible and structured grasps suitable, for
example, for a collaboration. Epic-Kitchens is another large audio-visual and
text dataset with a large variety of objects that are manipulated by the
person in the kitchen environment [9]. However, the dataset focuses on action
recognition from first person view and is not suitable for pose estimation as
it lacks the corresponding annotations. CORSMAL Containers Manipulation (CCM)
is a multi-view dataset consisting of videos of people manipulating various
household containers, e.g., pouring a content into a cup or drinking glass, or
shaking an already filled food box [13]. However, the number of instances for
each category is limited (15), and annotations of the object poses and
corresponding 3D models are not available. Despite all of these efforts, there
is still no large dataset for 6D pose estimation of objects that vary in their
physical properties and with different degrees of hand occlusions based on how
a person would grasp these objects.

Fig. 2: Sample of image crops from the generated mixed-reality dataset of
containers and hand-held containers. Note the diversity in human grasps,
colour and illumination on the rendered hand and forearms, and in object
texture and transparency.

In this paper, we present a new mixed-reality dataset for category-level pose
and size estimation of containers belonging to a set of categories (food
boxes, cups and drinking glasses) and held by a human hand. Fig. 2 shows
examples of image crops of the dataset. The dataset exhibits varying degrees
of occlusions and well-defined plausible grasp types (top, bottom, natural)
with the purpose of a collaboration [1]. Specifically, we generate the mixed-
reality dataset in a pseudo-realistic manner by rendering synthetic handheld
objects on top of real backgrounds, while accounting for their location in the
scene, and with plausible illumination, resembling the lights in the
background scene. We also provide an annotation of 60 sequences for each of
the three fixed views from CCM and show how our mixed-reality dataset enables
the training and deployment of deep learning models in real settings. To this
end, we validate an existing approach (i.e., NOCS [4]) on a test split of the
mixed-reality dataset and on the annotated real sequences111Data, models,
code, and additional results will be available at
https://corsmal.eecs.qmul.ac.uk/pose.html.

##  2 Mixed-reality dataset

In this section, we present our procedure to generate pseudo-realistic
composite images of handheld containers to form the mixed-reality dataset for
category-level object pose and size estimation in 3D.

We select containers that can be held or manipulated by a person, such as food
boxes and drinking glasses or cups, and of which the physical properties
highly vary in size, shape, transparency and textures (or absence of texture)
[13]. However, cups and drinking glasses differ only in their material, making
the recognition of these two types of objects hard for a vision model.
Therefore, we categorise the cups and drinking glasses based on their
geometry, and in particular by the presence of a stem. The categories are
human, non-stem, stem, box, and we include background for anything else [19].

As objects, we select 16 CAD models222Maximum number of unique stem-like
objects in ShapeNetSem. from ShapeNetSem with their corresponding textures for
each object category [20]. We scale the objects to realistic dimensions by
fixing the aspect ratio and sampling the height in the interval $[10,32]$ cm
for box, $[5,18]$ cm for non-stem, and $[6,20]$ cm for stem.

As grasp types, we follow the CORSMAL benchmark [1] and we define six ways to
hold an object based on the used hand and the position of the hand on the
object: grasp at the bottom with left hand; grasp on top with left hand;
natural grasp with left hand; grasp at the bottom with right hand; grasp on
top with right hand; and, natural grasp with right hand. Specifically, we use
the MANO hand model [21] and the GraspIt! [22] tool to manually generate
right-hand grasps for each of the 48 objects, and we mirror the right-hand
grasps to generate the left-hand versions. Because of the varying object
shapes and sizes, we manually annotate a total of 288 grasps.

As backgrounds, we acquire 30 images with an Intel D435i RealSense sensor in
10 different scenes (5 outdoor, 5 indoor) under 3 different views. Each
background contains a flat surface – e.g., a table or counter – where the
(handheld) objects are rendered. The sensor provides spatially aligned RGB and
depth images with a resolution of 640x480 pixels. Depth images are captured up
to a maximum distance of 6 m, and we apply spatial smoothing, temporal
smoothing, hole filling, and decimation as filters during acquisition. For
each background, we manually annotate a lighting setup using Blender [23] to
resemble the real scene conditions as close as we can when rendering the
object on top of the backgrounds. We use sun-like source light for outdoor
scenarios, whereas the lighting setup can include multiple omnidirectional and
directional point-like source lights and rectangle area-base lights for indoor
scenarios to reproduce bulb-lights, LEDs, and windows. These source lights are
adjusted in terms of position, orientation, colour and energy strength.

We generate a total of 138,240 composite images, split into 8,640 images of
only objects on top of flat surfaces and 129,600 image of handheld objects
above the flat surfaces. The dataset has 2,700 and 180 images for each object
with grasps and without grasp, respectively. Images are evenly distributed for
each grasp type (21,600), with 450 images for each object and grasp
combination.

For pseudo-realistically placing objects on top of flat surfaces in each
background scene, we manually segment the flat surface on the image plane [24]
and obtain the corresponding point cloud in 3D using the aligned depth map. We
then compute the normal of the surface by automatically segmenting the 3D
plane [25] and removing outliers [26]. After randomly sampling from a uniform
distribution a location on the flat surface where to place the object, we
align the vertical axis of the object with the normal of the flat surface in
the camera coordinate system and rotate the object around the vertical axis
with a rotation randomly sampled from a uniform distribution in the range
[0,359]∘. For each object and background combination, we sample 6 pairs of
locations and orientations and render the objects on top of the background
using Blender.

For pseudo-realistically rendering handheld objects, we place each object with
a MANO hand and a forearm from the SMPL+H model [21], based on the annotated
grasp type, at randomly sampled locations up to 40 cm above the segmented flat
surface for each background. We then rotate both the object and the hand by an
amount randomly sampled from a uniform distribution in the interval
$[-45,45]^{\circ}$ for pitch and roll, respectively. Note that for box, we
rotate the object of 180∘ around the vertical axis with a 50% chance to render
each side. To simulate a hand holding the object towards the camera, we place
the forearm pointing towards the camera principal and rotate the forearm with
an angle randomly sampled from a uniform distribution in the interval
$[-45,45]^{\circ}$ around the vertical axis. For each object and grasp type,
we sample 15 poses and render the handheld object on top of each background.
Note that when rendering the forearms and hands, we randomly change the skin
colour and the sleeves texture and type. We also avoid interpenetration
between the composite mesh (object and the hand holding the object) and the
flat surface by re-sampling another pose if any of the points belonging to the
flat surface has a Euclidean distance smaller than 0.5 mm with respect to the
nearest point on the composite mesh.

For each composite RGB image, we render accurate depth maps, segmentation
masks for the containers and hands, and maps of the normalised coordinates for
the containers [4], and we automatically annotate the 6D pose of the object.

##  3 Validation

In this section, we assess the benefit of our proposed mixed-reality dataset
for category-level 6D object pose estimation. To this end, we modify the
multi-branch deep learning model NOCS [4] to recognise our categories. We
compare the model that takes an RGB-D image as input with a modified post-
processing stage to estimate the 6D pose and size of the containers from an
RGB image with a prior.

###  3.1 Experimental setup

The original implementation of NOCS uses a ResNet-50 [27] as a backbone, as
there was no need to segment humans in their experimental settings. However,
features about the human category are learnt better at deeper layer by using a
ResNet-101 as in Mask R-CNN [19]. To obtain an accurate segmentation of both
the object and the person holding the object, we initialise the ResNet-101
backbone with pre-trained weights on COCO [28], and train the whole pipeline
using images from our mixed-reality dataset; 1721 images of cup, 1721 images
of wine glass, and 1721 images of human, from COCO 2017; and 1,721 images of
box from Open Images v6 [29]. Because of the different labelling, we map the
COCO classes cup and wine glass to our categories non-stem and stem,
respectively. Note that for Open Images we exclude images that contain people
due to the lack of annotated person segmentation masks. To balance the data
during training, we randomly sample 80% of images from our dataset and 20% of
images from COCO and Open Images together. When training the 3D coordinates
branch, the layers parameters are optimised only with the images and
annotations from the mixed-reality dataset.

Following the original training strategy [4], we train the model with their
multi-objective function that uses a cross-entropy loss for classification
[30], a smooth $\mathcal{L}_{1}$-norm loss for regressing the bounding box
[30], an average binary cross-entropy loss for binary segmentation [19] and a
symmetry-loss333Similar to NOCS [4], we select rotations with increments of
60∘ for non-stem and stem, and 180∘ for box around the vertical axis. for
estimating the 3D normalised coordinates [4]. For a single object, the total
loss is

$\mathcal{L}=\mathcal{L}_{\text{cls}}+\lambda_{1}\mathds{1}\mathcal{L}_{\text{bbox}}+\lambda_{2}\mathcal{L}_{\text{seg}}+\lambda_{3}\mathcal{L}_{\text{sym}},$
(1)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are hyper-parameters that control
the balance between the multiple losses [30], and the value of the indicator
function $\mathds{1}$ is 0 when the category is background.

NOCS uses Umeyama’s algorithm [31] to estimate the pose and size of the object
as the similarity transformation between the estimated 3D NOCS coordinates and
the 3D coordinates obtained from the depth map of the segmented object.
However, this assumes an accurate depth map that might not be available in the
presence of sensor noise or transparent objects, i.e., missing or noisy depth
values for the 3D point clouds can lead to wrong estimations of the object
poses. We observed that the predicted NOCS map intrinsically encodes the
projection of the 3D normalised object coordinates into the image of a camera
viewpoint and hence the pose of the object with respect to the canonical
representation. We exploit this relationship to directly estimate the object
pose up to a scale factor using EPnP [10] and the known camera intrinsic
parameters. The dimensions of the object in normalised space are obtained by
computing the absolute maximum after centring the object at the origin for
each coordinated axis. To recover the scaling factor, we consider and compare
three cases in addition to Umeyama’s algorithm: using a naive prior that is
the average scaling factor of a training object category selected by the NOCS
classification branch (EPnP-A); the real true scaling factor of the object as
a reference (EPnP-G); or the scaling factor estimated by Umeyama’s algorithm
(EPnP-U).

We use our mixed-reality dataset for training and testing the NOCS model. We
split the dataset into training, validation, and testing sets by leaving out
six instances for validation and six instances for testing (two instances per
category). This results in 103,680 images (36 instances) for training, 17,280
(6 instances) for validation, and 17,280 images (6 instances) for testing.

For testing generalisation, we consider CCM [13] and we select 60 sequences of
a person manipulating a container as observed from three fixed views (two side
and one frontal view). The sequences contain each of the fifteen containers
under four randomly sampled diverse conditions, including background and
lighting conditions, scenarios (person sitting, with the object on the table;
person sitting and already holding the object; person standing while holding
the container and then walking towards the table), and filling amount and
type. Fifteen sequences exhibit the case of the empty container for all
fifteen objects, whereas the other sequences have the person filling the
container with either pasta, rice or water at 50% or 90% of the full container
capacity. For each sequence, we manually annotate the 6D poses of the
containers every 10 frames if visible in at least two views [32]. We evaluate
the trained models with the four strategies on these annotated frames.

###  3.2 Performance measures

We evaluate the predicted poses and sizes using Average Precision:

$AP=\sum_{q=1}^{Q}(R_{q}-R_{q-1})\hat{P}(R_{q}),$ (2)

where $R_{1},\ldots,R_{Q}$ are the recalls in ascending order (detected
objects are sorted by 3D Intersection over Union (IoU) in descending order),
and $Q$ is the number of detected objects. $\hat{P}$ is the interpolated
precision at each recall $R$ by taking the maximum precision for any recall
level $\tilde{R}\geq R$: $\hat{P}(R)=\max_{\tilde{R}\geq R}P(\tilde{R})$,
where $P(\tilde{R})$ is the precision at recall $\tilde{R}$. The mean Average
Precision (mAP) is computed over the number of unique classes. We define a
true positive using the IoU, or Jaccard Index, between the predicted
($\tilde{b}$) and annotated 3D bounding box ($b$):
$J(\tilde{b},b)=(\tilde{b}\cap b)/(\tilde{b}\cup b)$, which satisfies the
condition $J(\tilde{b},b)\geq\tau$ (e.g. $\tau=50\%$). Note that we only
consider the detection with the correct class and highest IoU as a true
positive if there are multiple predictions above the threshold.

We evaluate object pose in 3D by computing AP for predictions whose
translation and rotation errors are less than a respective threshold [4]. The
translation error is defined as the Euclidean distance between the predicted
and annotated translation vectors,
$\varepsilon_{t}=\|\mathbf{\tilde{t}}-\mathbf{t}\|_{2}\in\mathbb{R}^{3}$. The
rotation error is given by
$\varepsilon_{R}=\arccos{((Tr(\mathbf{\tilde{R}}\mathbf{R}^{T})-1)/2)}$, where
$\tilde{\mathbf{R}}\in SO(3)$ ($\mathbf{R}$) is the predicted (annotated)
$3\times 3$ rotation matrix, $Tr(\cdot)$ is the trace operator, and T is the
transpose operator. Note that we convert $\varepsilon_{\mathbf{R}}$ from
radians to degrees for the discussion of the results. We define a true
positive as $\varepsilon_{\mathbf{t}}<\tau_{\mathbf{t}}$ and
$\varepsilon_{\mathbf{R}}<\tau_{\mathbf{R}}$.

Table 1: Average Precision results of the re-trained NOCS model using Umeyama’s algorithm for 3D object detection and 6D pose estimation per object category on the mixed-reality test set.  | $J_{25}$ | $J_{50}$ | $\varepsilon_{5,5}$ | $\varepsilon_{10,5}$ | $\varepsilon_{10,10}$ | $\varepsilon_{15,10}$
---|---|---|---|---|---|---
box | 99.3 | 92.9 | 2.9 | 24.7 | 24.7 | 53.3
nonstem | 62.9 | 61.6 | 10.8 | 36.3 | 36.3 | 58.3
stem | 85.8 | 84.3 | 9.7 | 34.1 | 34.2 | 53.3
Mean | 82.7 | 79.6 | 7.8 | 31.7 | 31.7 | 55.0

$0$$50$$100$$20$$40$$60$$80$$100$$\tau$mAP

$0$$30$$60$$\tau_{\mathbf{R}}$

$0$$25$$50$$\tau_{\mathbf{t}}$

$0$$50$$100$$20$$40$$60$$80$$100$$\tau$mAP

$0$$30$$60$$\tau_{\mathbf{R}}$

$0$$25$$50$$\tau_{\mathbf{t}}$

Fig. 3: Mean Average Precision (mAP) results while varying the threshold for
3D IoU ($\tau$, %), object orientation ($\tau_{\mathbf{R}}$, degrees) or
translation ($\tau_{\mathbf{t}}$, cm) on the mixed-reality test set (top) and
the CORSMAL Containers Manipulation dataset (bottom). Legend: EPnP-A ( ),
EPnP-G ( ), EPnP-U ( ), Umeyama ( ).

###  3.3 Results and discussion

We show the performance of the re-trained NOCS model with Umeyama for 3D
object detection and 6D pose estimation on the test set of our mixed-reality
dataset in Table 1. The model obtains a mAP of 55 percentage points (p.p.)
across the three categories with an error under 15∘ and 10 cm. This
performance decreases as we decrease the thresholds for orientation and
translation. This shows that it is still challenging for such a model to
achieve accurate pose estimations and correct object localisations, even on
composite images of a mixed-reality dataset and despite the presence of clean
and accurate depth maps. The category nonstem is the most affected among the
categories, especially due to false positives caused by inaccurate
localisation or misclassification. However, performance is not significantly
decreasing when increasing the threshold on the Jaccard index from 25 to 50.
On the contrary, pose performance is more affected with decreases of about 20
p.p. in mAP for a more restrictive threshold every 5∘ (e.g., from 15 to 5).
With the most restrictive threshold, $\varepsilon_{5,5}$, which would be an
appropriate error range for potential human-robot collaboration, we can
observe that the category box becomes the least accurate, whereas the model
retains 9.7% and 10.8% in mAP for stem and nonstem.

We compare the results of Umeyama, EPnP-A, EPnP-G, and EPnP-U, as we vary the
thresholds on the mixed-reality test set and on the annotated CCM sequences in
Fig. 3. Umeyama predicts more accurately poses and sizes than the EPnP
variants thanks to the clean depth in the mixed-reality dataset, but this
discrepancy reduces on CCM due to noisy depth images from the sensor and
presence of transparent containers. We noticed that hand-occlusions may cause
misclassification between nonstem and stem. When generalising to real data and
a new scene, it is hard for the model coupled with the pose recovery
strategies to preserve similar performance as in the mixed-reality test set.
This is shown by a drop in both translation and IoU as the thresholds are more
restrictive.

Fig. 4 shows sample results of 6D pose estimation on both testing datasets.
Note some large inaccuracies in the last two columns due to heavy occlusions
(small-size object), overestimated size, challenging object pose, or the gap
between synthetic and real data.

Fig. 4: Sample results (image crops) of 6D pose estimation with NOCS and
Umeyama on the mixed-reality test set (top) and with NOCS and EPnP-A on the
real sequences of CCM (bottom).

##  4 CONCLUSION

We presented a new mixed-reality dataset and annotated sequences of the
CORSMAL Containers Manipulation dataset [13] for the task of category-level 6D
pose estimation of hand-occluded containers. We showed that re-training a
model on our dataset improves the accuracy of poses and sizes of a set of
object categories. However, generalisation to a real scene is still
challenging due to noise in the depth data, transparencies and clutter. We
hope that the dataset and baseline model will facilitate research and
benchmarking.

Future work includes reducing the dependency on depth data and the application
of the model to human-robot collaboration.

## References

  * [1] R. Sanchez-Matilla, K. Chatzilygeroudis, A. Modas, N. Ferreira Duarte, A. Xompero, P. Frossard, A. Billard, and A. Cavallaro,  “Benchmark for human-to-robot handovers of unseen containers with unknown filling,”  IEEE Robotics Autom. Lett., vol. 5, no. 2, 2020.
  * [2] V. Ortenzi, A. Cosgun, T. Pardi, W. P. Chan, E. Croft, and D. Kulić,  “Object handovers: A review for robotics,”  IEEE Trans. Robotics, vol. 37, no. 6, pp. 1855–1873, Dec. 2021\.
  * [3] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield,  “Deep object pose estimation for semantic robotic grasping of household objects,”  in Proc. Conf. Robot Learn., 2018.
  * [4] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas,  “Normalized object coordinate space for category-level 6D object pose and size estimation,”  in Conf. Comput. Vis. Pattern Recognit., 2019.
  * [5] T. Hodaň, F. Michel, E. Brachmann, W. Kehl, A. Glent Buch, D. Kraft, B. Drost, J. Vidal, S. Ihrke, X. Zabulis, C. Sahin, F. Manhardt, F. Tombari, T-K. Kim, J. Matas, and C. Rother,  “BOP: Benchmark for 6D object pose estimation,”  in Eur. Conf. Comput. Vis., 2018.
  * [6] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic,  “CosyPose: Consistent multi-view multi-object 6D pose estimation,”  in Eur. Conf. Comput. Vis., 2020.
  * [7] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox,  “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes,”  in Robotics: Science and Syst., 2018.
  * [8] A. Ahmadyan, L. Zhang, J. Wei, A. Ablavatski, and M. Grundmann,  “Objectron: A large scale dataset of object-centric videos in the wild with pose annotations,”  in Conf. Comput. Vis. Pattern Recognit., 2021.
  * [9] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray,  “Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100,”  Int. J. Comput. Vis., vol. 130, pp. 33–55, 2022.
  * [10] V. Lepetit, F. Moreno-Noguer, and P. Fua,  “EPnP: An accurate O(n) solution to the PnP problem,”  Int. J. Comput. Vis., vol. 81, no. 2, pp. 155–166, 2009.
  * [11] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige,  “Keypose: Multi-view 3D labeling and keypoint estimation for transparent objects,”  in Conf. Comput. Vis. Pattern Recognit., 2020.
  * [12] M. Z. Irshad, S. Zakharov, R. Ambrus, T. Kollar, Z. Kira, and A. Gaidon,  “ShAPO: Implicit representations for multi object shape appearance and pose optimization,”  in Eur. Conf. Comput. Vis., 2022.
  * [13] A. Xompero, S. Donaher, V. Iashin, F. Palermo, G. Solak, C. Coppola, R. Ishikawa, Y. Nagao, R. Hachiuma, Q. Liu, F. Feng, C. Lan, R. H. M. Chan, G. Christmann, J. Song, G. Neeharika, C. K. T. Reddy, D. Jain, B. U. Rehman, and A. Cavallaro,  “The CORSMAL benchmark for the prediction of the properties of containers,”  IEEE Access, vol. 10, 2022.
  * [14] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab,  “Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes,”  in Asian Conf. Comput. Vis., 2012.
  * [15] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilic,  “Homebreweddb: RGB-D dataset for 6D pose estimation of 3D objects,”  in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2019.
  * [16] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit,  “HOnnotate: A method for 3D annotation of hand and object poses,”  in Conf. Comput. Vis. Pattern Recognit., 2020.
  * [17] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid,  “Learning joint reconstruction of hands and manipulated objects,”  in Conf. Comput. Vis. Pattern Recognit., 2019.
  * [18] D. Dwibedi, I. Misra, and M. Hebert,  “Cut, paste and learn: Surprisingly easy synthesis for instance detection,”  in Int. Conf. Comput. Vis., 2017.
  * [19] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick,  “Mask R-CNN,”  in Int. Conf. Comput. Vis., 2017.
  * [20] M. Savva, A. X. Chang, and P. Hanrahan,  “Semantically-enriched 3D models for common-sense knowledge,”  in Conf. Comput. Vis. Pattern Recognit. Workshops, 2015.
  * [21] J. Romero, D. Tzionas, and M. J. Black,  “Embodied hands: Modeling and capturing hands and bodies together,”  ACM Tran. Graphics, vol. 36, no. 6, pp. 245, 2017.
  * [22] A. T. Miller and P. K. Allen,  “GraspIt! a versatile simulator for robotic grasping,”  IEEE Robotics Autom. Magazine, vol. 11, no. 4, pp. 110–122, 2004\.
  * [23] Blender Online Community,  Blender - a 3D modelling and rendering package,  Blender Foundation, 2018.
  * [24] K. Wada,  “labelme: Image Polygonal Annotation with Python,” https://github.com/wkentaro/labelme, 2016.
  * [25] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus,  “Indoor segmentation and support inference from RGBD images,”  in Eur. Conf. Comput. Vis., 2012.
  * [26] M. A. Fischler and R. C. Bolles,  “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,”  Communic. ACM, vol. 24, no. 6, pp. 381–395, 1981.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Conf. Comput. Vis. Pattern Recognit., 2016.
  * [28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick,  “Microsoft COCO: Common objects in context,”  in Eur. Conf. Comput. Vis., 2018.
  * [29] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, T. Duerig, and V. Ferrari,  “The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale,”  Int. J. Comput. Vis., , no. 3, 2020.
  * [30] R. Girshick,  “Fast R-CNN,”  in Int. Conf. Comput. Vis., 2015.
  * [31] S. Umeyama,  “Least-squares estimation of transformation parameters between two point patterns,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 04, pp. 376–380, 1991.
  * [32] A. Xompero, Y. L. Pang, T. Patten, A. Prabhakar, B. Calli, and A. Cavallaro,  “Audio-visual object classification for human-robot collaboration,”  in Proc. IEEE Int. Conf. Acoustics, Speech Signal Process., 2022\.


# Lunar neutrinos

S. Demidov1,2, D. Gorbunov1,2
1 Institute for Nuclear Research of the Russian Academy of Sciences,
Moscow 117312, Russia
2 Moscow Institute of Physics and Technology,
Dolgoprudny 141700, Russia

###### Abstract

Cosmic rays bombard the lunar surface producing mesons, which attenuate inside
the regolith. They get slower and decay weakly into mostly sub-GeV neutrinos
leaving the surface. Thus the Moon shines in neutrinos. Here we calculate
spectra of low energy neutrinos, which exhibit bright features potentially
recognisable above isotropic neutrino background in the direction towards the
Moon. Their observation, though a very challenging task for future neutrino
large volume experiments, would make the Moon the nearest astrophysical source
for which the concept of multimessenger astronomy works truly. Remarkably,
some features of the lunar neutrino flux are sensitive to the surface mass
density of the Moon.

##  1 Introduction

So far we have observed only a single extra-terrestrial permanent neutrino
source, that is the Sun. Neutrinos can emerge upon hadron production in
astrophysical sources, see e.g. [1, 2], however the powerful ones are at quite
large distances, so the individual positional identification requires an
angular resolution far beyond the capabilities of the operating neutrino
telescopes. Naturally, on this way it is tempting to start with a close source
with obvious production mechanism to have both theoretical prediction robust
and positional identification reliable.

A promising source of this kind is the Moon. Cosmic rays (CR) hit its surface
freely, provided the absence of atmosphere. They initiate hadronic cascades
developing inside the regolith, and numerous mesons weakly decay producing
neutrinos. The cosmic ray spectrum degrades with energy and the scattering off
regolith slows down the mesons, so the largest neutrino flux is expected from
stopped mesons and muons. Hence the Moon becomes a source of neutrinos in sub-
GeV energy range to be observed on the sky by neutrino telescopes.
Semianalytic calculations of high energy (above 10 GeV) neutrino energy part
of the spectrum were performed in the study [3] (see also [4, 5] for earlier
studies). It showed a suppression of the lunar neutrino flux as compared to
the atmospheric one calculated over the Moon’s solid angle by factor of
$10^{-2}-10^{-4}$ depending on neutrino energy. Remarkably, both lunar and
atmospheric neutrinos have the same origin in cosmic rays.

In this letter we calculate the low energy part (from 10 MeV to about 10 GeV)
of lunar neutrino spectrum and compare the lunar neutrino flux at the Earth
with isotropic background from atmospheric and supernova neutrinos. We
numerically simulate interactions of cosmic rays with the regolith and count
neutrinos produced in meson and muon decays. We observe that although the
ratio of total lunar to atmospheric neutrino fluxes in the low energy range is
close to unity (contrary to the more energetic part) their spectra are quite
different. This observation makes it potentially possible to distinguish
between neutrinos of different origin, although the small Moon’s solid angle
with respect to the whole sky ($\sim 5\times 10^{-6}$) makes the task of lunar
neutrino detection a great challenge even for near future neutrino experiments
at the Earth.

Note that previous studies of the impact of cosmic ray bombarding the Moon
include in particular simulations of the production of cosmogenic nuclides [6]
and gamma-ray albedo [7]. Induced by the cosmic rays, $\gamma$-ray emission
from the Moon was measured by FERMI [8, 9]. A Moon shadow in high energy
cosmic rays was measured in numerous experiments, most recently by ANTARES
[10, 11] and IceCube [12, 13]. Possible implications of interactions of
astrophysical neutrinos in the Moon were discussed in [14].

##  2 Simulation details and results

Absence of atmosphere makes the Moon a very effective cosmic ray dump. Namely,
dominant sources of neutrinos, i.e. pions and kaons, which emerged in
collisions of cosmic rays with the Moon soil, stop before decay, hence
producing monoenergetic neutrinos.

To simulate neutrino flux from the Moon we exploit GEANT toolkit [15]. We
involve the cosmic ray spectra dominated by protons and 4He components. We fit
the corresponding plot from PDG [16] by a power-law function, see Fig. 1.

Figure 1:  Proton and 4He measured fluxes (the latter is scaled by $10^{-2}$
factor as in PDG) and numerical fits to them used in our analysis.

The fits look as (units are m-2s-1sr-1GeV-1)

$\displaystyle\frac{d^{2}\Phi}{dE_{kin}d\Omega}=\frac{2.7\times
10^{4}}{(E_{kin}+2.3m_{p})^{2.78}}\;\;\;\;\;\;\text{for protons}\,,$ (1)
$\displaystyle\frac{d^{2}\Phi}{dE_{kin}d\Omega}=\frac{1.5\times
10^{4}}{(E_{kin}+1.4m_{He})^{2.73}}\;\;\;\;\;\;\text{for ${}^{4}$He}\,,$ (2)

where $E_{kin}$ is kinetic energy of the nuclei, $m_{p}$ and $m_{He}$ are
masses of proton and 4He nuclei respectively. In simulations we set the
threshold kinetic energy $E_{kin}^{th}$ to $0.4$ GeV for protons and to 0.8
GeV for 4He. We checked that cosmic rays of lower energies produce rather
small amount of neutrinos.

The Moon chemical composition is taken as shown in Table 1

Element | Composition (wt%)
---|---
SiO2 | 45.5
Al2O3 | 19.5
CaO | 13.8
FeO | 10.0
MgO | 8.3
TiO2 | 2.3
Na2O | 0.6
Table 1: Chemical composition of lunar soil, adopted from Ref. [17].

which is an average of chemical compositions of Maria and Highlands parts of
the Moon surface, see e.g. [17]. For the lunar soil density we take
$\rho\approx 1.5$ g$/$cm3 which corresponds to an average density of the upper
layer of regolith. We assume the isotropic arrival directions of the cosmic
rays. We track the secondary particles produced by the collisions of cosmic
rays with the lunar surface using GEANT (we exploit FTFP_BERT physics model).
Within the simulation we count all electron and muon neutrinos and
antineutrinos leaving the Moon. Neutrino energy allows for constructing
spectra. The fluxes of $\tau$-neutrinos and antineutrinos at production are
negligible as compared to those emerged due to oscillations, which we discuss
below.

In Fig. 2 we show the obtained spectra of electron (left panel) and muon
(right panel) neutrinos

|
---|---
Figure 2:  Spectra of electron (left panel) and muon (right panel) neutrinos
and antineutrinos normalized to a single CR. The energy goes from 10 MeV to 10
GeV in decimal logarithmic scale with 320 bins.

and antineutrinos $N_{\nu}(E_{\nu})$ normalized to a single cosmic ray hitting
the Moon; we simulated about $1.5\times 10^{7}$ CR events and resulting
neutrino events are distributed over 320 uniform logarithmic energy bins. No
oscillations are taken into account at this stage. Most part of the neutrinos
come from charged pion and kaon decays as well as from decays of secondary
muons. Initially, pions $\pi^{+}$ and $\pi^{-}$ are produced in close
fractions. They loose their energies in the course of elastic collisions with
nuclei, interact and stop in the lunar soil. Most part of $\pi^{-}$ get
captured by nuclei via Coulomb attraction [18] and do not produce neutrinos
within discussed energy interval 10 MeV – 10 GeV. On the other hand, stopped
$\pi^{+}$ produce a monochromatic line of $\nu_{\mu}$ at energy
$E_{\nu}\approx 29.8$ MeV and anitmuons $\mu^{+}$ which also stop in media and
undergo the decay $\mu^{+}\to{\rm e}^{+}\nu_{e}\bar{\nu}_{\mu}$ yielding
neutrinos of energies below the threshold at $E_{\nu}\approx 52.8$ MeV.
Similar picture is valid for charged kaons, which produce a monochromatic
$\nu_{\mu}$ line at $E_{\nu}\approx 235.6$ MeV. On the right panel of Fig. 2
two peaks at $E_{\nu}\sim 30$ MeV and 236 MeV in the red histogram correspond
to those monochromatic neutrinos. The widths of 30 MeV and 236 MeV lines on
this plot are chosen to be equal to the bin width. Their actual widths are
considerably narrower, and hence the actual heights are considerably higher.
We clarify their contribution numerically in due course. There is also a small
bump at energies somewhat below the muon mass, $E_{\nu}\approx 100$ MeV. It
appears from the processes of muon capture by nuclei with subsequent
conversion to neutrino, i.e. $\mu^{-}+p\to n+\nu_{\mu}$, see e.g. [19].
Spectra of $\bar{\nu}_{\mu}$ and $\nu_{e}$ at
$E_{\nu}\mathchoice{\mathrel{\vbox{\hbox{$\buildrel\displaystyle<\over{\displaystyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\textstyle<\over{\textstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptstyle<\over{\scriptstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptscriptstyle<\over{\scriptscriptstyle\sim}$}}}}53$
MeV correspond to neutrinos from decays of antimuons $\mu^{+}$, most of which
decay at rest. Note in passing that neutrinos from stopped pions and kaons
were studied previously in different contexts (and in particular in searches
for a signal from dark matter annihilations in the Sun) in Refs. [20, 21, 22,
23, 24].

Now we turn to calculation of lunar neutrino flux at the Earth. The lunar
regolith emits neutrinos in all directions111Neglecting small cosmic ray
shadow by the Earth at the Moon.. Introducing total integrated over energy
isotropic flux $\Phi_{CR}$ of cosmic rays used in the simulations we can find
the number of cosmic ray particles bombarding the Moon per second

$\int 2\pi d(\cos{\theta})\cos{\theta}\,dS\Phi_{CR}=\pi\cdot 4\pi R_{\rm
Moon}^{2}\Phi_{CR}\,,$ (3)

where $R_{\rm Moon}$ is the Moon radius. Then, a neutrino $\nu_{\alpha}$ flux
at the Earth can be written as

$\Phi_{\nu_{\alpha}}=\frac{1}{4\pi L_{\rm Moon}^{2}}\times\pi\cdot 4\pi R_{\rm
Moon}^{2}\Phi_{CR}N_{\nu_{\alpha}}(E_{\nu})\equiv\pi\left(\frac{R_{\rm
Moon}}{L_{\rm Moon}}\right)^{2}\,\Phi_{CR}\,N_{\nu_{\alpha}}(E_{\nu})\,,$ (4)

where $L_{\rm Moon}$ is an average distance from the Earth to the Moon. Let us
note that given Moon’s orbit perigee about 362600 km and apogee about 405400
km one expects monthly variations of the lunar neutrino flux
$\Phi_{\nu_{\alpha}}$ with the amplitude about 12%.

In Figs. 3 and 4

|
---|---
Figure 3:  Fluxes of lunar neutrinos $\nu_{\mu}$ and antineutrinos $\bar{\nu}_{\mu}$ (red lines) in comparison with atmospheric (blue) and diffuse supernova (dashed black) neutrino fluxes calculated within a fraction of solid angle $\frac{\pi R_{\rm Moon}^{2}}{4\pi L_{\rm Moon}^{2}}$. No oscillations are taken into account. |
---|---
Figure 4:  Fluxes of lunar neutrinos $\nu_{e}$ and anitneutrinos
$\bar{\nu}_{e}$ (red lines) in comparison with atmospheric (blue) and diffuse
supernova (dashed black) neutrino fluxes calculated within a fraction of solid
angle $\frac{\pi R_{\rm Moon}^{2}}{4\pi L_{\rm Moon}^{2}}$. No oscillations
are taken into account.

we present differential neutrino fluxes multiplied by energy squared, i.e.
$E_{\nu}^{2}\frac{d\Phi_{\nu}}{dE_{\nu}}$ (in units of
m${}^{-2}\,$s${}^{-1}\,$GeV), for muon and electron (anti)neutrinos (red
lines) in comparison with atmospheric neutrino flux (blue lines) taken from
[25, 26] and diffuse supernova neutrino background (DSNB) flux (dashed black
lines) taken from [27] in the direction towards the Moon, which are
obtained222Let us note that that atmospheric neutrino flux depends on the
positioning of the detector, the difference is within a factor of two [25].
with multiplying the overall neutrino fluxes (no oscillations) by a factor
equal to the fraction of the celestial sphere occupied by the Moon,
$\frac{\Delta\Omega}{4\pi}=\frac{\pi R_{\rm Moon}^{2}}{4\pi L_{\rm
Moon}^{2}}\approx 5\times 10^{-6}$. Comparing, for instance, lunar and
atmospheric neutrinos in the energy interval from 10 MeV to 1 GeV we find that
ratio of their energy integrated fluxes within the same solid angle is close
to unity (about 1.4–1.5). At the same time, shapes of their energy spectra are
drastically different due to prominent features related to peculiarities of
neutrino production in the Moon. In particular, for an idealised neutrino
detector having energy resolution of 10% the fluxes of $\nu_{\mu}$ from the
Moon are about 0.14 and 0.0033 m-2s-1 for neutrino energies $29.8$ and 236
MeV, respectively, to be compared with atmospheric $\nu_{\mu}$ neutrino fluxes
0.0028 and 0.0034 m-2s-1, calculated over the solid angle of the Moon (without
oscillations).

As we discussed above interactions of parent mesons and muons in the media
have a great impact on spectra of neutrinos produced in collisions of cosmic
rays with the Moon surface. To illustrate this point we perform the same
numerical simulation taking somewhat larger values of the regolith density,
$\rho=1.95$ g$/$cm3. In Fig. 5

|
---|---
Figure 5:  Ratio of lunar neutrino flux at production calculated for different
regolith densities, 1.5 and 1.95 g/cm3, respectively.

we show the ratios of neutrino fluxes with densities 1.5 and 1.95 g$/$cm3,
respectively, for all neutrino flavors at production. We see the change of
density has the most prominent impact on $\nu_{\mu}$ flux. In particular, in
the matter of lower density the flux of $\nu_{\mu}$ is higher away from peaks
at 29.7 MeV and 236 MeV. This corresponds to an increase of probability for
mesons $\pi^{+}$ and $K^{+}$ (and subsequent $\mu^{+}$) to decay in flight. At
the same time, $\nu_{\mu}$ flux at the resonant energies, corresponding to
decays of mesons at rest, decreases. Also we observe an increase of
$\nu_{\mu}$ flux at the region around 100 MeV where these neutrinos appear
from $\mu^{-}-\nu_{\mu}$ conversion on nuclei. Fluxes of $\bar{\nu}_{e}$ are
also higher at energies corresponding to neutrino production from stopped
muons $\mu^{-}$. It is related to lower probability for $\pi^{-}$ to be
captured by nuclei in less dense media. At the same time fluxes of $\nu_{e}$
and $\bar{\nu}_{\mu}$ which come from decays of $\mu^{+}$ at this energy range
are almost unchanged. Note that we omit the part of the plot at larger
energies where small statistics do not allow to see clearly the density
impact. It is worth to note that most neutrinos in the interesting energy
range are produced in the surface upper layer of
$\mathchoice{\mathrel{\vbox{\hbox{$\buildrel\displaystyle<\over{\displaystyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\textstyle<\over{\textstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptstyle<\over{\scriptstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptscriptstyle<\over{\scriptscriptstyle\sim}$}}}}1$
m depth. One can also expect dependence of lunar neutrino flux as on the
density as well as on the composition of the regolith. Hence, the measurement
of the neutrino spectra provides with a tool to investigate the lunar surface
density.

Now let us study impact of neutrino oscillations on lunar neutrinos. As most
of them are produced in the surface region of the Moon, the corresponding
neutrino oscillation probabilities should be averaged over the production
point. The oscillation lengths for neutrinos of 10 MeV – 1 GeV is considerably
smaller than the baseline which is about $L_{\rm Moon}$ and for lower part of
this energy range is smaller than even radius of the Moon. This results in
almost incoherent neutrino flavour transitions for the softest part of lunar
neutrino spectra. However, matter effects in the Moon and the Earth may be
important333We are grateful to an anonymous referee for raising this
question.. To model neutrino oscillation in the Moon we use profile of lunar
density from Ref.[28] and we adopt PREM model [29] for the Earth structure to
describe neutrino oscillations in the Earth. We numerically solve the
Schrodinger equation for neutrino wave-function. For the PMNS matrix elements
we take the best fit to neutrino oscillation experiments from Ref. [30] for
normal neutrino mass hierarchy as an example. First, to demonstrate the impact
of the neutrino oscillation in matter of the Moon and the Earth we show in
Fig. 6

|
---|---
|
Figure 6:  Oscillation probabilities of lunar neutrinos at the Earth,
$P_{\nu_{e}\to\nu_{e}}$ (upper left), $P_{\bar{\nu}_{e}\to\bar{\nu}_{e}}$
(upper right), $P_{\nu_{\mu}\to\nu_{e}}$ (lower left),
$P_{\bar{\nu}_{\mu}\to\bar{\nu}_{e}}$ (lower right). Different curves
correspond to probabilities calculated 1) without matter effects; 2) with
matter effects in the Moon only; 3) with matter effects in the Moon and the
Earth. See main text for details.

neutrino oscillation probabilities $P_{\nu_{e}\to\nu_{e}}$ (upper left),
$P_{\bar{\nu}_{e}\to\bar{\nu}_{e}}$ (upper right), $P_{\nu_{\mu}\to\nu_{e}}$
(lower left), $P_{\bar{\nu}_{\mu}\to\bar{\nu}_{e}}$ (lower right) for three
different cases: 1) no matter effects in the Moon and the Earth; 2) with
matter effects in the Moon only; 3) matter effects in the Moon and the Earth.
In the latter case we consider neutrino propagating through the center of the
Earth, which implies in a real experiment that we account for the neutrino
signal only when the Moon is in Nadir. In all three cases we average over the
production point on the Moon surface and took $L_{Moon}=384000$ km as a
distance between the Earth and the Moon. We checked that variation of
$L_{Moon}$ due to ellipticity of the Moon orbit results only in minor
corrections to the oscillation probabilities. Also we average over neutrino
energy inside each of 320 energy bins. Second, in Fig. 7

|
---|---
|
Figure 7:  Oscillation probabilities to electron (anti)neutrinos at the Earth
for different zenith angles of the the Moon.

we show dependence of oscillation probabilities for electron (anti)neutrinos
on zenith angle $\theta_{M}$ of the Moon (the case $\cos{\theta_{M}}>0$
corresponds to the Moon above the horizon, and so no matter effect in the
Earth). In Fig. 8

|
---|---
|
Figure 8:  Oscillation probabilities for muon (anti)neutrinos at the Earth for
different zenith angles of the Moon.

we show the same probabilities but to muon (anti)neutrinos.

Impacts from matter effect in the Moon and the Earth results in deviations
from averaged neutrino oscillation probabilities and are clearly visible in
those Figures. However, see Sec. 3, to make these studies feasible, one needs
to collect a sufficient amount of lunar neutrinos from each position of the
Moon, which implies unrealistically large operation time of experiment. In
this case one may sum lunar neutrinos along the Moon trajectories, which
average the matter effect in the Earth, but keep intact the matter effect in
the Moon. In Fig. 9

|
---|---
|
Figure 9:  Fluxes of lunar neutrinos (left) and antineutrinos (right)
calculated with account of neutrino oscillations for $\cos{\theta_{M}}>0$
(upper panels) and after averaging over position of the Moon on the sky (lower
panels).

we show resulting spectra of lunar neutrinos and antineutrinos at the Earth
calculated for two cases: 1) assuming Moon to be above the horizon, which
implies no matter effect in the Earth; 2) averaging over position of the Moon
on the full sky. In realistic experimental setup additional averaging of
oscillations comes from finite energy and directional resolution. Detailed
study of this question is beyond the scope of this paper and generically
depends on the experimental parameters. We will comment on that in the next
Section. However, already from the plots of Fig. 9 one concludes that though
the difference is recognisable by eye, in a realistic experiment the matter
effect in the Earth has no observable effect at all.

Let us note in passing that although here we study the oscillations
numerically, one can use an analytical approach to describe evolution of low
energy neutrinos, see e.g. [31, 32]. Matter effect of neutrino propagation in
the Moon may be potentially used to obtain an information about its inner
structure similar to the studies for the Earth, see e.g. Refs. [33, 34, 35].

##  3 Discussion and conclusions

In the letter we calculated flux of neutrinos originated from cosmic rays
hitting the Moon surface. Apposed to the earlier study [3] we concentrated on
low energy
($\mathchoice{\mathrel{\vbox{\hbox{$\buildrel\displaystyle<\over{\displaystyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\textstyle<\over{\textstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptstyle<\over{\scriptstyle\sim}$}}}}{\mathrel{\vbox{\hbox{$\buildrel\scriptscriptstyle<\over{\scriptscriptstyle\sim}$}}}}10$
GeV) part of neutrino energy spectrum. Although lunar neutrinos have the same
origin as atmospheric ones, absence of atmosphere at the Moon makes spectrum
of the former to be quite peculiar and sensitive to the mass density of the
surface layer. In particular, it appears to be shifted to lower energies where
neutrinos originate from decays of mesons stopped in the regolith. Without
neutrino oscillations the very low energy part (less than about 52.8 MeV) of
the lunar neutrino spectrum at the Earth exceeds the atmospheric neutrino flux
from the Moon’s solid angle by up to an order of magnitude for $\nu_{e}$ and
$\bar{\nu}_{\mu}$ while the spectrum of $\nu_{\mu}$ exhibits two narrow peaks
at energies $E_{\nu}\approx 29.8$ MeV and 235.6 MeV corresponding to neutrinos
from $\pi^{+}$ and $K^{+}$ decays. Oscillations of neutrinos mix those effects
between all neutrino flavours. At energies lower than about 20–50 MeV the
lunar neutrino flux is comparable with that of DSNB within Moon’s solid angle.
Contrary to atmospheric and diffuse neutrinos the lunar neutrino flux is
expected to endue a periodic dependence with amplitude about 12% from small
ellipticity of the Moon orbit. These peculiarities, including explicit binding
to the direction of the Moon, can be used to search for lunar neutrinos in
future neutrino experiments and to distinguish them from atmospheric neutrino
and diffuse supernova neutrino background.

At present, neutrino detectors used or planned for study of sub-GeV neutrinos
include water Cerenkov (WC) detectors such as Hyper-Kamiokande [36, 37] and
its predecessor Super-Kamiokande, liquid scintillator (LS) and liquid argon
time projection chambers (LArTPC). The latter representative examples are JUNO
[38] and DUNE [39], respectively. WC detectors can be more easily realized in
large volume which is crucial requirement for detection of the pretty low
neutrino flux from the Moon. However, in neutrino-nucleon scattering
correlation between direction of charged lepton and incoming neutrino is weak
for neutrinos from sub-GeV energy range. Moreover, for such neutrino energies
an additional background in WC detectors comes from invisible muons. LS and
LArTPC detectors are free of this type of the background. Still, typical
detection channels used in LS detectors do not provide very large information
about neutrino direction in the considered energy range. LArTPC detectors on
the other hand can provide not only energy but also a directional information
(see e.g. [24]) as for them it is possible to reconstruct positions of charged
lepton and nucleon [40]. Still, even if very large exposures can be realized
here, at present an angular resolution of these detectors is still quite poor
as compared to the angular size of the Moon on the sky. We conclude that
detection of lunar neutrinos would require not large but huge neutrino
detectors with (so far) exceptional energy and, especially, angular resolution
– the task which may one day be feasible.

## Acknowledgements

We thank V. Izmodenov, A. Sanin, S. Troitsky for valuable discussions and Yu.
Kudenko, D. Naumov for clarification on the prospects of ongoing and future
neutrino experiments. This work is supported by the Ministry of science and
higher education of the Russian Federation under the contract #
075-15-2020-778 in the framework of the Large scientific projects program
within the national project ”Science”.

## References

  * [1] L. A. Anchordoqui et al., _Cosmic Neutrino Pevatrons: A Brand New Pathway to Astronomy, Astrophysics, and Particle Physics_ , _JHEAp_ 1-2 (2014) 1–30, [1312.6587].
  * [2] E. Vitagliano, I. Tamborra and G. Raffelt, _Grand Unified Neutrino Spectrum at Earth: Sources and Spectral Components_ , _Rev. Mod. Phys._ 92 (2020) 045006, [1910.11878].
  * [3] R. Miller and T. Cohen, _Cosmic-ray induced neutrino fluxes at the Moon: A semi-analytic approach_ , _Astropart. Phys._ 25 (2006) 368–374.
  * [4] L. V. Volkova and G. T. Zatsepin, _The energy spectra of muons and neutrinos generated by cosmic rays in different substances._ , in _International Cosmic Ray Conference_ , vol. 1 of _International Cosmic Ray Conference_ , p. 1039, Jan., 1965.
  * [5] L. V. Volkova, _Fluxes of Muons and Neutrinos Generated by Primary Radiation on the Moon_ , pp. 141–144.  Springer Netherlands, Dordrecht, 1989.
  * [6] Y. Li, X. Zhang, W. Dong, Z. Ren, T. Dong and A. Xu, _Simulation of the production rates of cosmogenic nuclides on the moon based on geant4_ , _Journal of Geophysical Research: Space Physics_ 122 (2017) 1473–1486.
  * [7] I. V. Moskalenko and T. A. Porter, _The Gamma-ray Albedo of the Moon_ , _Astrophys. J._ 670 (2007) 1467–1472, [0708.2742].
  * [8] A. Abdo et al., _FERMI observations of $\gamma$-ray emission from the Moon_, _Astrophys. J._ 758 (2012) 140.
  * [9] Fermi-LAT collaboration, M. Ackermann et al., _Measurement of the high-energy gamma-ray emission from the Moon with the Fermi Large Area Telescope_ , _Phys. Rev. D_ 93 (2016) 082001, [1604.03349].
  * [10] ANTARES collaboration, C. Distefano, _On the detection of the Moon shadow with the ANTARES neutrino telescope_ , _Nucl. Instrum. Meth. A_ 626-627 (2011) S223–S225.
  * [11] ANTARES collaboration, A. Albert et al., _The cosmic ray shadow of the Moon observed with the ANTARES neutrino telescope_ , _Eur. Phys. J. C_ 78 (2018) 1006, [1807.11815].
  * [12] IceCube collaboration, D. Boersma, L. Gladstone and A. Karle, _Moon Shadow Observation by IceCube_ , 1002.4900.
  * [13] IceCube collaboration, M. Aartsen et al., _Observation of the cosmic-ray shadow of the Moon with IceCube_ , _Phys. Rev. D_ 89 (2014) 102004, [1305.6811].
  * [14] D. Fargion, P. Oliva, P. G. de Sanctis Lucentini and M. Y. Khlopov, _Signals of HE atmospheric $\mu$ decay in flight around the Sun’s albedo versus astrophysical $\nu_{\mu}$ and $\nu_{\tau}$ traces in the Moon shadow_, _Int. J. Mod. Phys. D_ 27 (2018) 1841002, [1706.09352].
  * [15] GEANT4 collaboration, S. Agostinelli et al., _GEANT4–a simulation toolkit_ , _Nucl. Instrum. Meth. A_ 506 (2003) 250–303.
  * [16] Particle Data Group collaboration, M. Tanabashi et al., _Review of Particle Physics_ , _Phys. Rev. D_ 98 (2018) 030001.
  * [17] S. R. Taylor, _Lunar science: a post-Apollo view; scientific results and insights from the lunar samples._ , _New York: Pergamon Press_ (1975) .
  * [18] L. Ponomarev, _Molecular structure effects on atomic and nuclear capture of mesons_ , _Ann. Rev. Nucl. Part. Sci._ 23 (1973) 395–430.
  * [19] D. Measday, _The nuclear physics of muon capture_ , _Phys. Rept._ 354 (2001) 243–409.
  * [20] J. Spitz, _Cross Section Measurements with Monoenergetic Muon Neutrinos_ , _Phys. Rev. D_ 89 (2014) 073007, [1402.2284].
  * [21] C. Rott, J. Siegal-Gaskins and J. F. Beacom, _New Sensitivity to Solar WIMP Annihilation using Low-Energy Neutrinos_ , _Phys. Rev. D_ 88 (2013) 055005, [1208.0827].
  * [22] N. Bernal, J. Martin-Albo and S. Palomares-Ruiz, _A novel way of constraining WIMPs annihilations in the Sun: MeV neutrinos_ , _JCAP_ 08 (2013) 011, [1208.0834].
  * [23] C. Rott, S. In, J. Kumar and D. Yaylali, _Dark Matter Searches for Monoenergetic Neutrinos Arising from Stopped Meson Decay in the Sun_ , _JCAP_ 11 (2015) 039, [1510.00170].
  * [24] C. Rott, S. In, J. Kumar and D. Yaylali, _Directional Searches at DUNE for Sub-GeV Monoenergetic Neutrinos Arising from Dark Matter Annihilation in the Sun_ , _JCAP_ 01 (2017) 016, [1609.04876].
  * [25] G. Battistoni, A. Ferrari, T. Montaruli and P. Sala, _The atmospheric neutrino flux below 100-MeV: The FLUKA results_ , _Astropart. Phys._ 23 (2005) 526–534.
  * [26] M. Honda, M. Sajjad Athar, T. Kajita, K. Kasahara and S. Midorikawa, _Atmospheric neutrino flux calculation using the NRLMSISE-00 atmospheric model_ , _Phys. Rev. D_ 92 (2015) 023004, [1502.03916].
  * [27] C. A. J. O’Hare, A. M. Green, J. Billard, E. Figueroa-Feliciano and L. E. Strigari, _Readout strategies for directional dark matter detection beyond the neutrino background_ , _Phys. Rev. D_ 92 (2015) 063518, [1505.08061].
  * [28] J. G. Williams, A. S. Konopliv, D. H. Boggs, R. S. Park, D.-N. Yuan, F. G. Lemoine et al., _Lunar interior properties from the grail mission_ , _Journal of Geophysical Research: Planets_ 119 (2014) 1546–1578.
  * [29] A. M. Dziewonski and D. L. Anderson, _Preliminary reference earth model_ , _Physics of the Earth and Planetary Interiors_ 25 (1981) 297–356.
  * [30] I. Esteban, M. Gonzalez-Garcia, A. Hernandez-Cabezudo, M. Maltoni and T. Schwetz, _Global analysis of three-flavour neutrino oscillations: synergies and tensions in the determination of $\theta_{23}$, $\delta_{CP}$, and the mass ordering_, _JHEP_ 01 (2019) 106, [1811.05487].
  * [31] A. N. Ioannisian, N. A. Kazarian, A. Y. Smirnov and D. Wyler, _A Precise analytical description of the earth matter effect on oscillations of low energy neutrinos_ , _Phys. Rev. D_ 71 (2005) 033006, [hep-ph/0407138].
  * [32] O. L. G. Peres and A. Y. Smirnov, _Oscillations of very low energy atmospheric neutrinos_ , _Phys. Rev. D_ 79 (2009) 113002, [0903.5323].
  * [33] W. Winter, _Atmospheric Neutrino Oscillations for Earth Tomography_ , _Nucl. Phys. B_ 908 (2016) 250–267, [1511.05154].
  * [34] A. Ioannisian, A. Smirnov and D. Wyler, _Scanning the Earth with solar neutrinos and DUNE_ , _Phys. Rev. D_ 96 (2017) 036005, [1702.06097].
  * [35] P. Bakhti and A. Y. Smirnov, _Oscillation tomography of the Earth with solar neutrinos and future experiments_ , _Phys. Rev. D_ 101 (2020) 123031, [2001.08030].
  * [36] K. Abe et al., _Letter of Intent: The Hyper-Kamiokande Experiment — Detector Design and Physics Potential —_ , 1109.3262.
  * [37] Hyper-Kamiokande Proto- collaboration, K. Abe et al., _Physics potential of a long-baseline neutrino oscillation experiment using a J-PARC neutrino beam and Hyper-Kamiokande_ , _PTEP_ 2015 (2015) 053C02, [1502.05199].
  * [38] JUNO collaboration, F. An et al., _Neutrino Physics with JUNO_ , _J. Phys. G_ 43 (2016) 030401, [1507.05613].
  * [39] DUNE collaboration, R. Acciarri et al., _Long-Baseline Neutrino Facility (LBNF) and Deep Underground Neutrino Experiment (DUNE): Conceptual Design Report, Volume 2: The Physics Program for DUNE at LBNF_ , 1512.06148.
  * [40] C. Anderson et al., _The ArgoNeuT Detector in the NuMI Low-Energy beam line at Fermilab_ , _JINST_ 7 (2012) P10019, [1205.6747].


# Engagement Estimation in Advertisement Videos with EEG

Sangeetha Balasubramanian  International Institute of Information
TechnologyHyderabadIndia<EMAIL_ADDRESS>,
Shruti Shriya Gullapuram  University of Massachusetts AmherstAmherstUnited
States<EMAIL_ADDRESS>and  Abhinav Shukla  International Institute of
Information TechnologyHyderabadIndia<EMAIL_ADDRESS>
###### Abstract.

Engagement is a vital metric in the advertising industry and its automatic
estimation has huge commercial implications. This work presents a basic and
simple framework for engagement estimation using EEG (electroencephalography)
data specifically recorded while watching advertisement videos, and is meant
to be a first step in a promising line of research. The system combines recent
advances in low cost commercial Brain-Computer Interfaces with modeling user
engagement in response to advertisement videos. We achieve an F1 score of
nearly 0.7 for a binary classification of high and low values of self-reported
engagement from multiple users. This study illustrates the possibility of
seamless engagement measurement in the wild when interacting with media using
a non invasive and readily available commercial EEG device. Performing
engagement measurement via implicit tagging in this manner with a direct
feedback from physiological signals, thus requiring no additional human
effort, demonstrates a novel and potentially commercially relevant application
in the area of advertisement video analysis.

Engagement; Advertisements; EEG; Multimodal; Computational Advertising

††ccs: Human-centered computing HCI theory, concepts and models††ccs: Human-
centered computing User centered design

##  1\. Introduction

The popularity and versatility of social networks allows organizations to
reach their chosen target audience and offer products and services that suit
their constantly changing needs best. In this digital era, advertisements are
the best way to do so. Emotion and engagement play a powerful tool in
determining a person’s intent to buy a product. Positive and negative emotions
are used by advertisers to promote their product and drive users into buying
them. Brands associated with positivity has been shown to increase engagement.
However, negative emotion is also used to effectively garner consumer
attention, where certain life choices are portrayed as beneficial and
improving one’s quality of life, while others are portrayed as harmful and
potentially fatal. Advertisement valence (pleasantness), arousal (emotional
intensity) and engagement (emotional involvement) are key properties that play
a major role in consumer attitudes associated with the advertised product
(Holbrook and Shaughnessy, 1984; Holbrook et al., 1987; Pham et al., 2013).
Analyzing advertisements in terms of emotional content can also help optimize
user experience (Yadati et al., 2014).

This work expressly investigates the utility of physiological EEG
(electroencephalography) signals in the task of estimating engagement in
affective advertisement videos (evenly distributed over the emotional
dimensions). 14 channel EEG data is acquired via the commercial Emotiv headset
while the test subjects are watching the advertisements in question, and then
subsequently analyzed for engagement signatures. An implicit tagging based
approach is used to estimate the engagement level, without requiring any
additional effort from the end user.

In summary, we make the following contributions: 1) Our work is one of the
first to correlate user engagement in advertisement videos with the recorded
EEG signal 2) We demonstrate a simple yet effective way that a low cost
commercial EEG sensor can be employed for engagement recognition in the wild

Our paper is organized as follows. Section 2 discusses related work and
espouses the novelty of our work with respect to existing literature. Section
3 describes the ad dataset used in our experiments. Section 4 shows the EEG
data acquisition apparatus and protocol via the Emotiv headset. 5 discusses
experiments and key respective findings. Section 6 concludes the paper and
highlights important potential future implications and considerations.

Figure 1.  An EEG recording session with a subject

##  2\. Related Work

To highlight the novelty of our work, we briefly review prior works examining
(i) Conversational Systems, (ii) Engagement in online content, and (iii)
Online advertising.

###  2.1. Conversational Systems

Various multimodal features and machine learning algorithms are used to
predict engagement or involvement in human-human and human-robot conversations
(Yu et al., 2015; Celiktutan et al., 2017). Some work use more fine grained
features, such as Oertel et al. (Oertel and Salvi, 2013) used gaze features
obtained by an eye tracker to model engagement in multiparty conversations. In
more recent work (Yu, 2015), human behaviors, such as smiles and speech volume
was used to coordinate with users’ engagement on the fly via techniques such
as adaptive conversational strategies and incremental speech production.

Figure 2.  Representative thumbnail images from the videos in our dataset

###  2.2. Engagement in online content

Many researchers have analyzed engagement behavior towards web content.The
line of work that measures video engagement is often around measuring
engagement from the video popularity dynamics (number of views, likes, etc.)
(Park et al., 2016). More recent work (Wu et al., 2018) focuses on measuring
video engagement from public information, such as video context, topics,
number of times a video is watched and channel, without observing any user
reaction. While a lot of work is focused on engagement and attention in online
web content and videos, there is very little work determining the engagement
in advertisement videos.

###  2.3. Online advertising

Advertising is most effective when it elicits an emotional response and
engages the viewer on a personal level. There has been extensive study in the
context of display advertising (Azimi et al., 2012; Rosales et al., 2012) and
sponsored search (Becker et al., 2009b; Sculley et al., 2009; Sodomka et al.,
2013). A lot of work focused on predicting performance of an ad based on its
click-through rate. Later efforts has been on matching the web queries or
pages to the context of the ad (Becker et al., 2009b, a; Lee et al., 2013;
Oentaryo et al., 2014). More recent work (Barbieri et al., 2016) focuses on
analyzing the level of user engagement with ads as the length of time a user
spends after landing on an ad page. There have also been recent attempts to
use EEG to mine affective attributes from data (Shukla et al., 2017b) and use
it for more emotionally relevant advertising. A very recent work also uses
contextual and visual attention data from eye tracking to suggest emotionally
driven design principles for advertisements (Shukla et al., 2018).

However, we specifically study the potential contribution of the physiological
EEG modality to engagement estimation. We use an emotional advertisement
dataset curated by (Shukla et al., 2017a), the details of which follow in the
next section. These ads are then presented as stimuli to test subjects whose
EEG signal data is recorded. The subjects also self-report engagement on a
5-point scale. Engagement estimation is achieved via EEG signals acquired via
the wireless and wearable Emotiv headset, which facilitates naturalistic user
behavior and can be employed for large-scale engagement recognition from
multimedia content.

##  3\. Advertisement Dataset Description

This section presents details regarding the ad dataset used in this study..

We use an affective advertisement dataset curated by (Shukla et al., 2017a)
for our study. Defining valence as the degree of pleasantness/unpleasantness
and arousal as the intensity of emotional feeling, five experts carefully
compiled a dataset of 100, roughly 1-minute long commercial advertisements
(ads). These ads are publicly available111On video hosting websites such as
YouTube. The authors of (Shukla et al., 2017a) chose the ads based on
consensus among five experts on valence and arousal labels (either high
(H)/low (L)). High valence ads typically involved product promotions, while
low valence ads were social messages depicting ill effects of smoking, alcohol
and drug abuse.

Figure 3.  Engagement rating distribution with Gaussian pdf overlay Table 1.  Mean correlations between self-rated attributes. Significant correlations ($p<0.05$) are denoted in bold. | Arousal | Valence | Engagement
---|---|---|---
Arousal | 1 | -0.19 | 0.36
Valence |  | 1 | 0.11
Engagement |  |  | 1

To evaluate the effectiveness of these ads as affective control stimuli,
(Shukla et al., 2017a) examined how consistently they could evoke target
emotions across viewers. To this end, the ads were independently rated by 14
annotators for valence (valence) and arousal in (Shukla et al., 2017a). All
ads were rated on a 5-point scale, which ranged from -2 (very unpleasant) to 2
(very pleasant) for valence and 0 (calm) to 4 (highly aroused) for arousal.

In this work, we chose a similar setting for the rating of engagement.
Defining engagement as the emotional involvement or commitment while viewing
an audio-visual stimulus on a 5-point scale from 0 (least engaging) to 4 (most
engaging) as shown in Fig. 4, we employed 23 annotators for the engagement
rating task, with each person viewing a variable number of ads. For each
advertisement, we calculated the mean of ratings of all annotators and then
thresholded this average by the grand average of all ratings of all ads to get
a binary label that we treated as ground truth in all our engagement
estimation experiments.

The distribution of engagement ratings (Fig. 3) is roughly uniform resulting
in a Gaussian fit with large variance, with the mean observed at the median
scale value of 2.

Pearson correlation was computed between the arousal, valence and engagement
(see table 1) dimensions by limiting the false discovery rate to within 5%
(Benjamini and Hochberg, 1995). This procedure revealed a weak and
insignificant negative correlation of 0.17, implying that ad arousal and
valence scores were largely uncorrelated. However, we found that arousal and
engagement had a statistically significant positive correlation of 0.36, which
goes to show that advertisements with high affective activation tend to be
more engaging.

##  4\. EEG Acquisition Protocol

Figure 4.  Rating prompt shown to the subject after each stimulus
advertisement

As the annotators rated the ads for engagement upon watching them, we acquired
their Electroencephalogram (EEG) brain activations via the Emotiv wireless
headset as shown in Fig. 1. To maximize attention and minimize fatigue during
the rating task, these raters took a break after every 20 ads, and viewed the
entire set of 100 ads over five sessions (some users viewed slightly lesser
ads). Upon viewing each ad, the raters had a maximum of 10 seconds to input
their engagement scores via mouse clicks using a prompt shown in Fig 4. The
Emotiv device comprises of 14 electrodes (locations shown in Fig 5), and has a
sampling rate of 128 Hz. Upon experiment completion, the EEG recordings were
segmented into epochs, with each epoch denoting the viewing of a particular
ad. Upon removal of noisy epochs. We recorded a total of 1738 epochs. Each ad
was preceded by a 1s fixation cross to orient user attention, and to measure
resting state EEG power used for baseline power subtraction. The EEG signal
was band-limited between 0.1–45 Hz, and independent component analysis (ICA)
was performed to remove artifacts relating to eye movements, eye blinks and
muscle movements. The following section describes the techniques employed for
content-centered AR and user-centered AR.

Figure 5.  The electrode locations of the 14 channel Emotiv EEG headset

##  5\. Experiments and Results

Table 2. Advertisement Engagement Prediction from EEG analysis. F1 scores are presented in the form $\mu\pm\sigma$.  Method | Engagement
---|---
| F1 for first 30 s (F30) | F1 for last 30 s (L30) | F1 for last 10 s (L10)
LDA | 0.6954 $\pm$ 0.0207 | 0.6781 $\pm$ 0.0256 | 0.6644 $\pm$ 0.0222
LSVM | 0.7018 $\pm$ 0.0201 | 0.6888 $\pm$ 0.0217 | 0.6647 $\pm$ 0.0211
RSVM | 0.7091 $\pm$ 0.0224 | 0.6803 $\pm$ 0.0204 | 0.6732 $\pm$ 0.0263

We first provide a brief description of the classifiers used and settings
employed for engagement estimation, where the objective is to assign a binary
(high/low) label for engagement evoked by each ad, using the extracted EEG
features. The ground truth here is provided by the mean of annotator ratings
for each ad followed by thresholding by the grand mean of all ad ratings.
Experimental results will be discussed hereafter.

#### Classifiers:

We employed the Linear Discriminant Analysis (LDA), linear SVM (LSVM) and
Radial Basis SVM (RSVM) classifiers in our AR experiments. LDA and LSVM
separate high/low labeled training data with a hyperplane, while RSVM is a
non-linear classifier which separates high and low classes, linearly
inseparable in the input space, via transformation onto a high-dimensional
feature space. We notice that RSVM classifier produces the best F1-scores for
engagement.

#### Metrics and Experimental Settings:

We used the F1-score (F1), defined as the harmonic average of the precision
and recall as our performance metric, due to the unbalanced distribution of
positive and negative samples.

The 1738 clean epochs obtained from the EEG was used for user-centered
analysis. To maintain dimensional consistency for subsequent principal
component analysis (PCA), we performed user-centric AR experiments with (a)
the first 3667 samples (30s of EEG data), (b) the last 3667 samples (30s of
EEG data) and (c) the last 1280 samples (10s of EEG data) from each epoch.
Each epoch sample comprises data from 14 EEG channels, and the epoch samples
were input to the classifier upon vectorization and dimensionality reduction
of the time-domain EEG data by PCA.

In this work, we use only time-domain EEG information. As we evaluate
engagement performance on a small dataset, AR results obtained over 10
repetitions of 5-fold cross validation (CV) (total of 50 runs) are presented.
CV is typically used to overcome the overfitting problem on small datasets,
and the optimal SVM parameters are determined from the range
$[10^{-3},10^{3}]$ via an inner five-fold CV on the training set. Finally, in
order to examine the temporal variance in AR performance, we present F1-scores
obtained over (a) first 30s (F30), (b) last 30s (L30) and (c) last 10s (L10).
These settings were chosen to study the efficiency with which engagement
prediction can be realized at different temporal segments of the video to
study whether there is a potential accuracy drop-off over time.

###  5.1. Discussion

Table 2 summarizes the results that we get for the various experiments for
engagement estimation. The best performance that we achieve among the tested
methods is using the RSVM classifier for the first 30 seconds of the ads (F1 =
0.709). RSVM is also uniformly the best classifier as compared to both LDA and
LSVM, which points to the possibility that the decision boundary for
engagement related statistics in EEG data is non-linear.

From the observations we infer that engagement recognition is lower in the L30
and L10 conditions, with a clear drop in accuracy later in the video. This
could be because of a drop in engagement levels towards the end of the
advertisement where strictly product related information is more prevalent as
opposed to an engaging storyline in the beginning.

The observations also highlight the limitation of using a single engagement
label for the whole video (as opposed to dynamic labeling).

##  6\. Conclusion and Future Work

This work presents a first step towards using EEG data to directly predict the
engagement level of a user while watching a video advertisement. There
certainly exist many limitations that can be worked on in the future. Some of
these are:-

  * •

The level of engagement is treated here as a binary high or low label only. A
much better and more rigorous way of doing this would be to annotate a
continuous level of emotion at every time point using an annotation tool like
Feeltrace (Cowie et al., 2000) and a regression to estimate the continuous
value instead of a categorical classification. However, such an annotation is
very time consuming and has issues to deal with like synchronization.

  * •

This study only deals with the EEG modality. Combining it with other
modalities that could be useful in a similar environment such as facial
expressions, eye tracking, GSR, or even audiovisual content analysis would be
an interesting avenue to explore.

  * •

This study only considers conventional classifiers (LDA and SVM). It would be
interesting to see how the recent advances in deep learning (especially using
transfer learning for small dataset sizes) can work on such a problem.

  * •

This study only considers the raw time domain EEG data that is cleaned using
ICA and subsequently vectorized, dimensionality reduced, and then passed to a
classifier. A lot of EEG related literature suggests that spectral features
from the alpha, beta, gamma and theta bands are useful for EEG classification.
However, these conventional features did not work as well for us and the time
domain summary statistics using PCA gave us the best performance.

However, despite these limitations, the findings of this study offer an
important insight into how a portable and commercially viable EEG device may
be used for engagement measurement in the wild. This can have important
industrial applications in the field of ad impact analysis. There exist modern
tools by companies like Realeyes (rea, 2018) such as Creative Testing that
provide second-by-second insights into advertisement engagement levels. They
are based on principles of implicit tagging (Pantic and Vinciarelli, 2009;
Soleymani et al., 2012) that are based on the user not having to consciously
annotate any content in the video but their implicit responses (via EEG in our
case) being able to automatically deduce the required value. The low cost and
portability of the EEG headset used in the study further illustrates that the
prototypical engagement estimation system can be subsequently developed into a
viable commercial product.

## References

  * (1)
  * rea (2018) 2018\.  RealEyes Emotional Intelligence.  https://www.realeyesit.com/. (2018).
  * Azimi et al. (2012) Javad Azimi, Ruofei Zhang, Yang Zhou, Vidhya Navalpakkam, Jianchang Mao, and Xiaoli Fern. 2012\.  Visual appearance of display ads and its effect on click through rate. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 495–504.
  * Barbieri et al. (2016) Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. 2016\.  Improving post-click user engagement on native ads via survival analysis. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 761–770.
  * Becker et al. (2009a) Hila Becker, Andrei Broder, Evgeniy Gabrilovich, Vanja Josifovski, and Bo Pang. 2009a.  Context transfer in search advertising. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, 656–657.
  * Becker et al. (2009b) Hila Becker, Andrei Broder, Evgeniy Gabrilovich, Vanja Josifovski, and Bo Pang. 2009b.  What happens after an ad click?: quantifying the impact of landing pages in web advertising. In Proceedings of the 18th ACM conference on Information and knowledge management. ACM, 57–66.
  * Benjamini and Hochberg (1995) Yoav Benjamini and Yosef Hochberg. 1995.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.  J. Royal Stat. Soc. Series B (Methodological) 57, 1 (1995), 289–300.
  * Celiktutan et al. (2017) Oya Celiktutan, Efstratios Skordos, and Hatice Gunes. 2017\.  Multimodal human-human-robot interactions (mhhri) dataset for studying personality and engagement.  IEEE Transactions on Affective Computing (2017).
  * Cowie et al. (2000) Roddy Cowie, Ellen Douglas-Cowie, Susie Savvidou*, Edelle McMahon, Martin Sawey, and Marc Schröder. 2000.  ’FEELTRACE’: An instrument for recording perceived emotion in real time. In ISCA tutorial and research workshop (ITRW) on speech and emotion.
  * Holbrook et al. (1987) Morris B Holbrook, Rajeev Batra, and Rajeev Batra. 1987\.  Assessing the Role of Emotions as Mediators of Consumer Responses to Advertising.  Journal of Consumer Research 14, 3 (1987), 404–420.  DOI:https://doi.org/10.1002/job.305
  * Holbrook and Shaughnessy (1984) Morris B Holbrook and John O Shaughnessy. 1984.  The role of emotlon in advertising.  Psychology & Marketing 1, 2 (1984), 45–64.
  * Lee et al. (2013) Jung-Hyun Lee, Jongwoo Ha, Jin-Yong Jung, and Sangkeun Lee. 2013\.  Semantic contextual advertising based on the open directory project.  ACM Transactions on the Web (TWEB) 7, 4 (2013), 24\.
  * Oentaryo et al. (2014) Richard J Oentaryo, Ee-Peng Lim, Jia-Wei Low, David Lo, and Michael Finegold. 2014\.  Predicting response in mobile advertising with hierarchical importance-aware factorization machine. In Proceedings of the 7th ACM international conference on Web search and data mining. ACM, 123–132.
  * Oertel and Salvi (2013) Catharine Oertel and Giampiero Salvi. 2013.  A gaze-based method for relating group involvement to individual engagement in multimodal multiparty dialogue. In Proceedings of the 15th ACM on International conference on multimodal interaction. ACM, 99–106.
  * Pantic and Vinciarelli (2009) Maja Pantic and Alessandro Vinciarelli. 2009.  Implicit human-centered tagging [Social Sciences].  IEEE Signal Processing Magazine 26, 6 (2009).
  * Park et al. (2016) Minsu Park, Mor Naaman, and Jonah Berger. 2016.  A Data-Driven Study of View Duration on YouTube.. In ICWSM. 651–654.
  * Pham et al. (2013) Michel Tuan Pham, Maggie Geuens, and Patrick De Pelsmacker. 2013\.  The influence of ad-evoked feelings on brand evaluations: Empirical generalizations from consumer responses to more than 1000 {TV} commercials.  International Journal of Research in Marketing 30, 4 (2013), 383 – 394.
  * Rosales et al. (2012) Rómer Rosales, Haibin Cheng, and Eren Manavoglu. 2012\.  Post-click conversion modeling and analysis for non-guaranteed delivery display advertising. In Proceedings of the fifth ACM international conference on Web search and data mining. ACM, 293–302.
  * Sculley et al. (2009) D Sculley, Robert G Malkin, Sugato Basu, and Roberto J Bayardo. 2009.  Predicting bounce rates in sponsored search advertisements. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1325–1334.
  * Shukla et al. (2017a) Abhinav Shukla, Shruti Shriya Gullapuram, Harish Katti, Karthik Yadati, Mohan Kankanhalli, and Ramanathan Subramanian. 2017a.  Affect Recognition in Ads with Application to Computational Advertising. In Proceedings of the 2017 ACM on Multimedia Conference (MM ’17). ACM, New York, NY, USA, 1148–1156.  DOI:https://doi.org/10.1145/3123266.3123444
  * Shukla et al. (2017b) Abhinav Shukla, Shruti Shriya Gullapuram, Harish Katti, Karthik Yadati, Mohan Kankanhalli, and Ramanathan Subramanian. 2017b.  Evaluating Content-centric vs. User-centric Ad Affect Recognition. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 402–410.  DOI:https://doi.org/10.1145/3136755.3136796
  * Shukla et al. (2018) Abhinav Shukla, Harish Katti, Mohan Kankanhalli, and Ramanathan Subramanian. 2018.  Looking Beyond a Clever Narrative: Visual Context and Attention are Primary Drivers of Affect in Video Advertisements. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM.
  * Sodomka et al. (2013) Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. 2013.  A predictive model for advertiser value-per-click in sponsored search. In Proceedings of the 22nd international conference on World Wide Web. ACM, 1179–1190.
  * Soleymani et al. (2012) Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. 2012.  A multimodal database for affect recognition and implicit tagging.  IEEE Transactions on Affective Computing 3, 1 (2012), 42–55.
  * Wu et al. (2018) Siqi Wu, Marian-Andrei Rizoiu, and Lexing Xie. 2018\.  Beyond Views: Measuring and Predicting Engagement in Online Videos.  (2018).
  * Yadati et al. (2014) Karthik Yadati, Harish Katti, and Mohan Kankanhalli. 2014\.  CAVVA: Computational affective video-in-video advertising.  IEEE Trans. Multimedia 16, 1 (2014), 15–23.
  * Yu (2015) Zhou Yu. 2015.  Attention and engagement aware multimodal conversational systems. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 593–597.
  * Yu et al. (2015) Zhou Yu, Alexandros Papangelis, and Alexander Rudnicky. 2015\.  TickTock: A non-goal-oriented multimodal dialog system with engagement awareness. In Proceedings of the AAAI Spring Symposium, Vol. 100.