∎

11institutetext: 1 Faculty of Information Technology, University of Science,
Ho Chi Minh City, Vietnam, 11email<EMAIL_ADDRESS>
2 Vietnam National University, Ho Chi Minh City, Vietnam
3 Department of Computer Science, University of Liverpool, UK, 11email:
<EMAIL_ADDRESS>
∗ Corresponding author

# Global-Local Attention for Emotion Recognition

Nhat Le1,2 Khanh Nguyen1,2 Anh Nguyen3,∗ Bac Le1,2

(Received: date / Accepted: date)

###### Abstract

Human emotion recognition is an active research area in artificial
intelligence and has made substantial progress over the past few years. Many
recent works mainly focus on facial regions to infer human affection, while
the surrounding context information is not effectively utilized. In this
paper, we proposed a new deep network to effectively recognize human emotions
using a novel global-local attention mechanism. Our network is designed to
extract features from both facial and context regions independently, then
learn them together using the attention module. In this way, both the facial
and contextual information is used to infer human emotions, therefore
enhancing the discrimination of the classifier. The intensive experiments show
that our method surpasses the current state-of-the-art methods on recent
emotion datasets by a fair margin. Qualitatively, our global-local attention
module can extract more meaningful attention maps than previous methods. The
source code and trained model of our network are available at
https://github.com/minhnhatvt/glamor-net.

###### Keywords:

Emotion recognition Facial expression recognition Attention Deep network

††journal: Neural Computing and Applications

##  1 Introduction

Emotion recognition aims to classify input data into several expressions that
convey universal emotions, such as angry, disgust, fear, happy, neutral, sad,
and surprise. The input data can be one or more of different modalities such
as visual information, audio, and text 8014813  han2014speech  7374704 . Due
to the availability of a large number of images and videos on the Internet,
inferring human emotion from visual content, is considered to be one of the
most popular tasks. Recently, automatic emotion recognition has gained a lot
of attention in both academia and industry 6940284 . It enables a wide range
of novel applications in different domains, ranging from healthcare
do2021multiple , surveillance Author02  to robotics nguyen2020autonomous  and
human-computer interaction Author03 .

|
---|---
|

(a)

(b)

(c)

(d)

Figure 1: Examples of human emotion detection results from our method

Traditional methods for emotion recognition combine handcrafted features (e.g,
histogram of oriented gradients (HOG) hog , local binary patterns) with
classifiers such as SVM svm  or graphical models graphical_models . With the
popularity of deep learning techniques, especially Convolutional Neural
Network (CNN) cnn , together with the exists of many large-scale datasets, the
meaningful features can be extracted using a deep network. However, the
majority of previous methods onlyface1 onlyface2  onlyface3 FERpriormotion
only exploit features from human’s face, and use this information to predict
human emotions. These works assume that the facial region is the most
informative representation of human emotion, therefore they ignore the
surrounding context, which is shown to play an important role in the
understanding of the perceived emotion, especially when the emotions on the
face are expressed weakly or indistinguishable caer .

Recently, researchers have been focusing on incorporating background
information such as people’s pose, gaits, etc., into the model to improve the
performance emoticon binh2021graph . In this work, we follow the same
direction. However, unlike other works that learn the facial and context
information independently emoticon , we propose to jointly learn both facial
and context information using our new Global-Local Attention mechanism. We
hypothesize that the local information (i.e., facial region) and global
information (i.e., context background) have a correlative relationship, and by
simultaneously learning the attention using both of them, the accuracy of the
network can be improved. This is based on the fact that the emotion of one
person can be indicated by not only the face’s emotion (i.e., local
information) but also other context information such as the gesture, pose, or
emotion/pose of a nearby person. Fig 1 shows some recognition results of our
proposed method.

To verify the effectiveness of our approach, we benchmark on the CAER-S
dataset caer , a large-scale dataset for context-aware emotion recognition. We
achieved $77.90\%$ top-1 accuracy on the test set, which is an improvement of
$4.38\%$ over the recent state-of-the-art method caer . Furthermore, with the
integrated ResNet-18 resnet  as the backbone network, we obtained state-of-
the-art performance on the CAER-S dataset with $\mathbf{89.88}\%$
classification accuracy. We also present a novel way to create a new static-
image dataset from videos of the CAER dataset caer . The experiments on this
new dataset also confirm that our proposed method consistently achieves better
performance than previous state-of-the-art approaches.

In summary, our contributions are as follows:

  * •

We propose a new deep network, namely, Global-Local Attention for Emotion
Recognition Network (GLAMOR-Net) that surpasses the state-of-the-art methods
in the emotion recognition task.

  * •

In GLAMOR-Net, we proposed the Global-Local Attention module, which
successfully encodes both local features from facial regions and global
features from surrounding background to improve the human emotion
classification accuracy.

  * •

We perform extensive experiments to validate the effectiveness of our proposed
method and the contribution of each module on recent challenging datasets.

The paper is organized as follow: We review the related work in Section 2. We
then describe our methodology in detail in Section 3. In Section 4, we present
extensive experimental results on challenging datasets and analyze the
contribution of each module in GLAMOR-Net. Finally, we conclude the paper and
discuss future work in Section 5.

##  2 Related Work

###  2.1 Human Emotion

In the late twentieth century, Ekman and Friesen discovered six basic
universal emotions including anger, disgust, fear, happiness, sadness, and
surprise 6emotions . Several years later, contempt was added and considered as
one of the basic emotions contempt . However, our affective displays in
reality are much more complicated and subtle compared to the simplicity of
these universal emotions. To represent the complexity of the emotional
spectrum, many approaches were proposed such as the Facial Action Coding
System facs , where all facial actions are described in terms of Action Units
(AUs); or dimensional models dim_model , where affection is quantified by
values chosen over continuous emotional scales like valence and arousal.
Nevertheless, those models which use discrete affections are the most popular
in automatic emotion recognition task because they are easier to interpret and
more intuitive to human.

###  2.2 Emotion Recognition

In automatic human emotion recognition, many approaches mainly focus on
analyzing facial expression. Thus, a standard emotion recognition system
usually consists of three main stages: face detection, feature extraction and
expression classification onlyface1 onlyface2 onlyface3 FERpriormotion ).
Traditional methods relied on handcrafted features (LBPLBP , HOGhog ) to
extract meaningful features from input images, and classifiers (such as SVM or
random forest) to classify human emotions based on extracted features. With
the rise of deep learning, CNN-based methods have made significant progress in
the task of emotion recognition deepfer_survey . Apart from using input image,
other works focus on categorizing emotions by utilizing extra information such
as speech speech_emotion1 speech_emotion2 , human pose body_emotion1 , body
movements and gaits gait_emotion1 gait_emotion2 . However, these works have
relied on the information coming from a single modality, hence they have
limited ability to fully exploit all usable information of human emotions.

Figure 2: The architecture of our proposed network. The whole process includes
three steps. First, we extract the facial information (local) and context
information (global) using two Encoding Modules. Second, we feed the extracted
face and context features into the Global-Local Attention (GLA) module to
perform attention inference on the global context. Lastly, we fuse both
features from the facial region and output features from GLA into a neural
network to make final emotion classification

To overcome this limitation, many researches have investigated the use of
multiple modalities. Primarily, these works tried to fuse multiple channels of
information from each modality to predict emotion. Castellano _et al._
multimodal1  used extracted features from three different modalities (facial
expressions, body gestures and speech expressions), and then fused those
modalities at two different levels (i.e. feature level and decision level).
Their results showed that the fusion performed at the feature level provided
better results than the one performed at the decision level. Sikka _et al._
multimodal2  extracted different visual features such as SIFT-Bag of Words
sift_bow , LPQ-TOP lpq_top , HOG hog_features , PHOG phog , and GIST gist  and
fuse them with audio features by building a kernel from each set of features,
then combine them using a SVM classifier. Likewise, the authors in multimodal3
used the same multi-modality approach but using deep learning techniques. In
emoticon , three interpretations of context information are fused together by
a deep neural network to classify human emotions in an end-to-end manner.

Recently, many works have focused on exploring context-aware information for
emotion recognition. Kosti _et al._ context_emotic  and Lee _et al._ caer
proposed two architectures based on deep neural networks for learning context
information. Both of them have two separate branches for extracting different
kinds of information. One branch focuses on human features (i.e. face for caer
and body for context_emotic ) and the other concentrates on surrounding
context. When considering multiple modalities, which have a large amount of
information, deep learning-based methods like emoticon caer context_emotic
do2018affordancenet  are more suitable and effective than traditional
approaches. These multi-modal approaches often yield better classification
performance than uni-modal methods.

###  2.3 Attention Model

Attention was first introduced in machine translation machinetranslation ,
allowing the translation model to search for words in the input sentence that
are more relevant to the prediction words. Since then, attention models have
become an important concept and an essential component of neural network
architectures. It has made significant impacts in many application domains,
including natural language processing NLPattention , computer vision
CVattention , graph Graphattention  and speech processing Speechattention .

In emotion recognition, attention models were mainly used to discover the
attentive areas of the face that need to be focused on FERpriormotion .
Recently, the work that forced the model to pay attention to the most
discriminative regions of the background using attention was proposed in CAER-
Net-S caer . However, previous work only used the background encoding to learn
the context saliency map and did not take advantage of the facial
representation to assist the process. Therefore, we propose the Global-Local
Attention mechanism, which takes both facial and context encoding as inputs,
to utilize facial information more efficiently to guide the context saliency
map learning procedure.

##  3 Methodology

###  3.1 Overview

In this work, we assume that emotions can be recognized by understanding the
context components of the scene together with the facial expression. Our
method aims to do emotion recognition in the wild by incorporating both facial
information of the person’s face and contextual information surrounding that
person. Our model consists of three components: Encoding Module, Global-Local
Attention (GLA) Module, and Fusion Module. Our main contribution is the novel
GLA module, which utilizes facial features as the local information to attend
better to salient locations in the global context. Fig. 2 shows an overview of
our method.

Figure 3: Our proposed encoder network as the feature extractor for both face
and context branches. The network contains five convolutional layers with ReLU
non-linearity, each convolution is followed by a max pooling layer except the
last one to reduce the spatial dimensions of the input

###  3.2 Network Architecture

####  3.2.1 Encoding Module

To detect human emotion, many works first process the image by cropping out
the human faces from the scene, and then feed them into a convolutional
network to extract facially-expressive features onlyface1 onlyface2 onlyface3
FERpriormotion . We generally follow this approach in our Encoding Module. In
particular, our Encoding Module comprises the Facial Encoding Module to learn
the face features, and the Context Encoding Module to learn the context
features.

Facial Encoding Module This module aims to learn meaningful features from the
facial region of the input image. The facial embedding information can be
denoted as $\mathbf{F}_{f}$:

$\mathbf{F}_{f}=\mathfrak{C}(\mathbf{I}_{f};\theta_{f})$ (1)

where $\mathfrak{C}$ is the convolutional operation parameterized by
$\theta_{f}$, and $\mathbf{I}_{f}$ is the input facial region. In practice, we
use a sub-network (Fig. 3) as the feature extractor for the Facial Encoding
Module.

The proposed sub-network has five convolutional layers. Particularly, each
convolutional layer has a kernel set of $3\times 3$ filters with strides of
$1\times 1$ followed by a Batch Normalization layer and a ReLU activation
function. The number of filters starts with 32 in the first layer, increasing
by a factor of 2 at each subsequent layer except the last one. Our network
ends up with 256 output channels. We also use the padding technique before
each convolutional layer to keep the output spatial dimensions the same as the
input. The output of each convolutional layer is pooled using a max-pooling
layer with strides of $2\times 2$. The encoding module outputs a 256-channel
volume feature map, which is the embedded representation with respect to the
input image.

Context Encoding Module This module is used to exploit background knowledge to
support the emotion predicting process. Similar to the Facial Encoding Module,
we follow the same procedure to extract context information contained in the
scene with a different set of parameters:

$\mathbf{F}_{c}=\mathfrak{C}(\mathbf{I}_{c};\theta_{c})$ (2)

where $\mathfrak{C}$ is the convolutional operation parameterized by
$\theta_{c}$, and $\mathbf{I}_{c}$ is the input context. Similar to the Facial
Encoding Module, we use the sub-network (Fig. 3) to extract deep features from
the background context region in the Context Encoding Module.

After getting these two feature maps, we feed them into the Global-Local
Attention Module to calculate the attention scores for regions in the context.
However, if we extract the context information in the raw image where the
faces apparently exist, the network will also encode the facial information.
This problem can make the attention module produce trivial outputs because the
network may only focus on the facial region, and omitting the context
information in other parts of the image. To address this problem, we first
detect the face and then hide it in the raw input by setting all the values in
the facial region to zero.

####  3.2.2 Global-Local Attention Module

Figure 4: The proposed Global-Local Attention module takes the extracted face
feature vector and the context feature map as the input to perform context
attention inference. Each vector $\mathbf{v}_{i,j}$ in the context feature map
$\mathbf{F}_{c}$ is concatenated with the face vector $\mathbf{v}_{f}$ and
then fed into a sub-network to compute the attention weight for the $(i,j)$
position. The final output vector is a linear combination of all regions in
the context weighted by the corresponding attention weight. For efficiency,
our attention inference network contains a 128-unit Fully Connected layer with
the ReLU activation function and a Softmax layer. Weights are shared across
all the context regions Figure 5: The Fusion Module consists of two separate
sub-networks, each network computes the fusion weights for face branch and
context branch. The input vector of each branch is then scaled by its
corresponding weight and combined together into the final representation
vector $\mathbf{v}$. We use this vector $\mathbf{v}$ to estimate the emotion
category by feeding it into another sub-network (see Fig. 2)

Inspired by the attention mechanism Speechattention  nguyen2019v2cnet , to
model the associative relationship of the local information (i.e., the facial
region in our work) and global information (i.e., the surrounding context
background), we propose the Global-Local Attention Module to guide the network
focus on meaningful regions (Fig. 4). Specifically, our attention mechanism
models the hidden correlation between the face and different regions in the
context by capturing their similarity using deep learning techniques. Our
attention module takes the extracted face feature map $\mathbf{F}_{f}$ and the
context feature map $\mathbf{F}_{c}$ from the two encoding modules as input,
and then outputs a normalized saliency map that has the same spatial dimension
as $\mathbf{F}_{c}$.

In practice, we first reduce the facial feature map $\mathbf{F}_{f}$ into
vector representation using the Global Pooling operator, denoted as
$\mathbf{v}_{f}$. Note that the context feature map $\mathbf{F}_{c}$ is a 3D
tensor, $\mathbf{F}_{c}\in\mathbb{R}^{H_{c}\times W_{c}\times D_{c}}$, where
$H_{c}$, $W_{c}$, and $D_{c}$ are the height, width, and channel dimension
respectively. We derive the context feature map $\mathbf{F}_{c}$ as a set of
$W_{c}*H_{c}$ vectors with $D_{c}$ dimensions, each vector in each cell
$(i,j)$ represents the embedded features at that location, which can be
projected back to the corresponding patch in the input image:

$\mathbf{F}_{c}=\\{\mathbf{v}_{i,j}\in\mathbb{R}^{D_{c}}|1\leq i\leq
H_{c},1\leq j\leq W_{c}\\}$ (3)

At each location $(i,j)$ in the context feature map, we have
$\mathbf{F}_{c}^{(i,j)}=\mathbf{v}_{i,j}$, where
$\mathbf{v}_{i,j}\in\mathbb{R}^{D_{c}}$ and $1\leq i\leq H_{c}$, $1\leq j\leq
W_{c}$.

We concatenate $[\mathbf{v}_{f};\mathbf{v}_{i,j}]$ into a big vector
$\bar{\mathbf{v}}_{i,j}$, which contains both information about the face and
some small regions of the scene. We then employ a feed-forward neural network
to compute the score corresponding to that region by feeding
$\bar{\mathbf{v}}_{i,j}$ into the network. After repeating the same process
for all regions, each region $(i,j)$ will output a raw score value $s_{i,j}$,
we spatially apply the Softmax function to produce the attention map:

$a_{i,j}=\frac{\exp(s_{i,j})}{\Sigma_{a}\Sigma_{b}\exp(s_{a,b})}$ (4)

To obtain the final context representation vector, we squish the feature maps
by taking the average over all the regions weighted by $a_{i,j}$ as follow:

$\mathbf{v}_{c}=\Sigma_{i}\Sigma_{j}(a_{i,j}\odot\mathbf{v}_{i,j})$ (5)

where $\mathbf{v}_{c}\in\mathbb{R}^{D_{c}}$ is the final single vector
encoding the context information, and $\odot$ is the scalar multiplication
operation. Additionally, $\mathbf{v}_{c}$ mainly contains information from
regions that have high attention, while other unimportant parts of the context
are mostly ignored. With this design, our attention module can guide the
network focus on important areas based on both facial information and context
information of the image. Note that, in practice, we only need to extract
context information once and then using different encoded face representations
to make the system look at different regions with respect to that person.

####  3.2.3 Fusion Module

The Fusion Module is used to incorporate the facial and context information
more effectively when predicting human emotions. The Fusion Module takes
$\mathbf{v}_{f}$ and $\mathbf{v}_{c}$ as the input, then the face score and
context score are computed independently by two neural networks:

$\displaystyle s_{f}=\mathcal{F}(\mathbf{v}_{f};\phi_{f})$ $\displaystyle
s_{c}=\mathcal{F}(\mathbf{v}_{c};\phi_{c})$ (6)

where ${\phi_{f}}$ and $\phi_{c}$ are the network parameters of the face
branch and context branch, respectively. Next, we normalize those scores by
the Softmax function to produce weights for each face and context branch so
that these weights sum up to 1.

$\displaystyle w_{f}=\frac{\exp(s_{f})}{\exp(s_{f})+\exp(s_{c})}$
$\displaystyle w_{c}=\frac{\exp(s_{c})}{\exp(s_{f})+\exp(s_{c})}$ (7)

|
---|---

(a) CAER

(b) NCAER-S

Figure 6: Percentage of each emotion category in the CAER and the new NCAER-S
training sets

Notice that the face weight and the context weight are independently computed
by their corresponding networks and represent the importance of these
branches. We let the two networks competitively determine which branch is more
useful than the other. Then we amplify the more useful branch and lower the
effect of the other by multiplying the extracted features with the
corresponding weight:

$\displaystyle\mathbf{v}_{f}\leftarrow\mathbf{v}_{f}\odot w_{f}$
$\displaystyle\mathbf{v}_{c}\leftarrow\mathbf{v}_{c}\odot w_{c}$ (8)

Finally, we use these vectors to estimate the emotion category. Specifically,
in our experiments, after multiplying both $\mathbf{v}_{f}$ and
$\mathbf{v}_{c}$ by their corresponding weights, we concatenate them together
as the input for a network to make final predictions. Fig. 5 shows our fusion
procedure in detail.

##  4 Experiments

###  4.1 Datasets

CAER-S In this work, we only focus on static images with background context as
our input. Therefore, we choose the static CAER (CAER-S) dataset caer  to
validate our method. The CAER-S dataset contains 70K static images extracted
from a total of 13201 video clips of 79 TV shows. Each image is labeled with
one of seven universal emotions: anger, fear, disgust, happiness, neutral,
sadness and surprise. We follow the standard split proposed by caer  for
training, validation and testing, respectively.

Novel CAER-S (NCAER-S) While experimenting with the CAER-S dataset, we observe
that there is a correlation between images in the training and the test sets,
which can make the model less robust to changes in data and may not generalize
well on unseen samples. More specifically, many images in the training and the
test set of the CAER-S dataset are extracted from the same video, hence making
them look very similar to each other. To cope with this issue, we propose a
novel way to extract static frames from the CAER video clips to create a new
static image dataset called Novel CAER-S (NCAER-S). In particular, frames
extracted from the training, validation, and test sets of the CAER dataset are
separately put into the corresponding training, validation, and test sets of
the new NCAER-S dataset. In particular, for each video in the original CAER
dataset, we split the video into multiple parts, each part is approximately
$2$s long. Then we randomly select one frame of each part to include in the
new NCAER-S dataset. Any original video that provides frames for the training
set will be removed from the testing set. This process assures the new dataset
is novel while the training frames and testing frames are never from one
original input video.

With our selection method, we ensure that images in the validation and test
sets are independent of those in the training set. We also make sure that the
numbers of extracted frames of each emotion category are approximately equal
to tackle the imbalance problem of the CAER dataset and prevent bias towards
prominent emotions.

The statistics of the original CAER and the new NCAER-S training sets are
shown in Fig. 6 and Table 1. The new split NCAER-S dataset can be downloaded
at https://bit.ly/NCAERS_dataset.

Table 1: The number of images in each emotion category in the NCAER-S training set Emotion | Number of images
---|---
Angry | 2272
Disgust | 3004
Fear | 2902
Happy | 1905
Neutral | 3202
Sad | 3084
Surprise | 2186
Total | 18555

###  4.2 Experimental Setup

Evaluation Metric. Classification accuracy is the standard evaluation metric
that is widely used to measure the reliability of automated emotion
recognition systems in the literature deepfer_survey caer ck_plus afew
affectnet . To compare our results with previous approaches quantitatively, as
in deepfer_survey caer  we use the overall classification accuracy as the
evaluation metric:

$\text{Accuracy}=\frac{1}{N}{\Sigma_{i=1}^{N}\mathds{1}\\{\hat{y}_{i}=y_{i}\\}}$
(9)

where $\mathds{1}$ is the indicator function, $N$ is the total number of
samples in the dataset, $\hat{y}_{i}$ and $y_{i}$ is the network prediction
and ground-truth category of the $i$-th example, respectively.

Baselines. We compare the results of our proposed Global-Local Attention for
Emotion Recognition network (GLAMOR-Net) with the following methods as
baselines: AlexNet alexnet , VGGNet vgg , ResNet resnet , CAER-Net-Scaer . The
results of AlexNet, VGGNet, and ResNet on the CAER-S dataset are reported in
two cases: using the ImageNet dataset as the pre-trained model, and fine-
tuning these networks on this dataset. Note that, these results are taken from
caer  paper. On the CAER-S, we also compare our method to several recent
state-of-the-art approaches. GRERN GRERN  utilized a multi-layer Graph
Convolutional Network (GCN) to exploit the relationship among different
regions in the context. EfficientFace EfficientFace  proposed an efficient
lightweight network and utilized the label distribution to handle the
ambiguity of real-world emotions. MA-Net manet  designed a highly complicated
architecture based on ensemble learning of multiple regions to handle the
occlusion and pose variation problems. We report the results of our GLAMOR-Net
with two different backbones: the original encoding module introduced in
section 3.2.1 and ResNet-18 resnet .

Implementation Details. Our networks are implemented using Tensorflow 2.0
framework tf . For optimization, we use the SGD optimization algorithm and
standard cross-entropy loss function:

$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\log p_{i}^{(y_{i})}$ (10)

where $p_{i}^{(y_{i})}$ is the predicted probability for the true emotion
category $y_{i}$ of the $i$-th sample and $N$ is the total number of samples
in the dataset.

Given an input image, we first use the CNN based face detector in the dlib
library dlib  to detect the face coordinates. The detected face is then
cropped and resized to 96 x 96 and fed to the Facial Encoding Module. To
create input for the Context Encoding Module, we mask the facial region in the
original image and resize it to 128 x 171, then we apply random crop during
the training phase and center crop during the inference phase to the final
size of 112 x 112. We use a dropout layer before the final layer with a
dropout rate of 0.5 to reduce the effect of overfitting. During training, we
observe that the fusion network is very unstable and easily affected by random
factors. Specifically, the weights of the face branch or the context branch in
the Fusion Module can easily take a value near 0 or 1, which means the model
completely ignores information extracted from one of the branches. To tackle
this problem, we first train the Facial Encoding Module and the Context
Encoding Module separately, then jointly train both modules and the fusion
network in an end-to-end manner.

###  4.3 Results

####  4.3.1 Results on the CAER-S dataset

Table 2 summarizes the results of our network and other recent state-of-the-
art methods on the CAER-S dataset caer . This table clearly shows that
integrating our GLA module can significantly improve the accuracy performance
of the recent CAER-Net. In particular, our GLAMOR-Net (original) achieves
77.90% accuracy, which is a +4.38% improvement over the CAER-Net-S. When
compared with other recent state-of-the-art approaches, the table clearly
demonstrates that our GLAMOR-Net (ResNet-18) outperforms all those methods and
achieves a new state-of-the-art performance with an accuracy of 89.88%. This
result confirms our global-local attention mechanism can effectively encode
both facial information and context information to improve the human emotion
classification results.

Fig 7 shows the confusion matrix of the GLAMOR-Net (original) on the CAER-S
dataset. Overall, the model achieves the highest accuracy on the fear class
with $0.96$ accuracy. The neutral class has the lowest accuracy of $0.63$ as
there are many misclassifications from other classes.

Table 2: Classification accuracy of baseline methods and our GLAMOR-Net on the CAER-S dataset Methods | Year | Accuracy (%)
---|---|---
ImageNet-AlexNet alexnet  | 2012 | 47.36
ImageNet-VGGNet vgg  | 2015 | 49.89
ImageNet-ResNet resnet  | 2016 | 57.33
Fine-tuned AlexNet alexnet  | 2012 | 61.73
Fine-tuned VGGNet vgg  | 2015 | 64.85
Fine-tuned ResNet resnet  | 2016 | 68.46
CAER-Net-S caer  | 2019 | 73.52
GRERN GRERN  | 2020 | 81.31
EfficientFace EfficientFace  | 2021 | 81.48
MA-Net manet  | 2021 | 88.42
GLAMOR-Net (original) | 2021 | 77.90
GLAMOR-Net (ResNet-18) | 2021 | 89.88
Figure 7: The confusion matrix of our GLAMOR-Net (original) results on the
CAER-S test set

####  4.3.2 Results on the NCAER-S dataset

On the NCAER-S dataset, we compare our results with three recent methods:
VGG16 vgg , ResNet50 resnet , and CAER-Net-S caer . The results from the VGG16
and ResNet50 models are reproduced as baseline methods. We finetune the VGG16
and the ResNet50 from the pre-trained models on VGG-Face and ImageNet,
respectively. Our GLAMOR-Net (original) and CAER-Net-S are trained from
scratch for a fair comparison.

Table 3: Classification accuracy of baseline methods and our GLAMOR-Net on the NCAER-S dataset Methods | Accuracy (%)
---|---
VGG16 vgg  | 42.85
ResNet50 resnet  | 41.41
CAER-Net-S caer  | 44.14
GLAMOR-Net (original) | 46.91
Figure 8: The confusion matrix of our GLAMOR-Net (original) results on the
NCAER-S test set

Table 3 reports the comparative results of our GLAMOR-Net and other recent
methods. This table shows that the GLAMOR-Net architecture outperforms all
other architectures and achieves the highest performance. In particular, our
network increases classification accuracy by 2.77% compared to the second-
highest model CAER-Net-S. These results also validate the effectiveness of our
proposed global-local attention mechanism integrated into the GLAMOR-Net. We
note that the result of VGG16 pre-trained on VGG-Face is surprisingly better
than the result of ResNet50 pre-trained on ImageNet dataset. This is
explainable as the pre-trained weight on VGG-Face carries more meaningful
information than the pre-trained weight on ImageNet, which includes many non-
face images.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s) (a) Anger

(t) (b) Disgust

(u) (c) Fear

(v) (d) Happy

(w) (e) Sad

(x) (f) Surprise

Figure 9: Visualization of the attention maps. From top to bottom: original
image in the NCAER-S dataset, image with masked face, attention map of the
CAER-Net-S, and attention map of our GLAMOR-Net

Also from Table 3, we can see that the classification accuracy of the models
is much lower than those in Table 2. The reason behind this is the new NCAER-S
is more challenging than the original CAER-S dataset. As mentioned earlier, to
construct the NCAER-S dataset, we eliminate the correlation between the train
and the test samples as much as we can. Specifically, we separately resample
image frames from clips of the train and test sets of the CAER dataset to
mitigate the train and test dependency. Moreover, note that the size of the
new dataset is only less than one-third of the original one, which also limits
the amount of information that the models can exploit. However, our GLAMOR-Net
still consistently outperforms other state-of-the-art methods despite the
challenges of the NCAER-S dataset and shows competitive results.

The confusion matrix of our GLAMOR-Net evaluated on the NCAER-S dataset is
given in Fig 8. The two categories with the highest accuracy are happy and
neutral while the disgust emotion has the lowest accuracy of 0.28. It can also
be inferred from the confusion matrix that our model mostly confuses neutral
with other emotion categories as most of the misclassified examples of the six
categories: angry, disgust, fear, happy, sad and surprise fall into the class
neutral. It may be because the facial emotion in the NCAER-S dataset is weakly
expressed, which makes it more difficult to identify and distinguish other
emotions from the neutral class.

In summary, we can conclude that our method consistently improves the results
on both the original CAER-S and the challenging NCAER-S datasets. Note that
although we follow the same procedure as in caer , our proposed Global-Local
Attention Module is the key difference that helps enhance the accuracy of the
emotion recognition task. The results reported in Table 2 and Table 3 verify
that with the assistance of our attention strategy, the classification
accuracy is significantly improved. We believe that if a more sophisticated
neural architecture is adopted, the performance will be further boosted.

####  4.3.3 Analysis

Table 4: Ablation study of our proposed method on the NCAER-S dataset. ‘w/F’,
‘w/mC’, ‘w/fC’, ‘w/CA’, ‘w/GLA’ denote using the output of the Facial Encoding
Module, the Context Encoding Module with masked faces as input, the Context
Encoding Module with visible faces as input, the standard Context Attention in
CAER-Net-S caer  and our Global-Local Attention Module, respectively, as input
to the Fusion Network

Settings | w/F | w/mC | w/fC | w/CA | w/GLA | Accuracy (%)
---|---|---|---|---|---|---
(i) | ✓ |  |  |  |  | 42.58
(ii) |  | ✓ |  |  |  | 41.18
| ✓ |  | ✓ |  | 41.27
| ✓ |  |  | ✓ | 42.24
(iii) |  |  | ✓ | ✓ |  | 41.94
|  | ✓ |  | ✓ | 42.66
(iv) | ✓ | ✓ |  |  |  | 43.19
✓ | ✓ |  | ✓ |  | 44.14
✓ | ✓ |  |  | ✓ | 46.91

Table 5: $p$ value of the Stuart-Maxwell test for each pair of methods that are used in the setting (iv) of Table 4. Each element on the main diagonal is the test result of the agreement between the model prediction and the observed data (ground-truth label). Methods | w/GLA | w/CA | w/o Attention
---|---|---|---
w/GLA | $3.0\times 10^{-2}$ | $1.69\times 10^{-14}$ | $1.38\times 10^{-53}$
w/CA |  | $1.33\times 10^{-10}$ | $3.33\times 10^{-14}$
w/o Attention |  |  | $2.81\times 10^{-38}$

To further analyze the contribution of each component in our proposed method,
we experiment with 4 different input settings on the NCAER-S dataset: (i) face
only, (ii) context only with the facial region being masked, (iii) context
only with the facial region visible, and (iv) both face and context (with
masked face). When the context information is used, we compare the performance
of the model with different context attention approaches (no attention,
standard attention module in CAER-Net-S and our GLA module). Note that to
compute the saliency map with the proposed GLA in the (ii) and (iii) setting,
we extract facial features using the Facial Encoding Module, however, these
features are only used as the input of the GLA module to guide the context
attention map learning process and not as the input of the Fusion Network to
predict the emotion category. The performances of these settings are
summarized in Table 4.

The results clearly show that our GLA consistently helps improve performance
in all settings. Specifically, in setting (ii), using our GLA achieves an
improvement of 1.06% over method without attention, 0.97% over standard
attention module in CAER-Net-S caer . It is also noteworthy that when the
context with visible faces is utilized as in setting (iii), using the
attention module in the CAER-Net-S achieves 41.94% accuracy, lower than the
one using only the cropped face in setting (i) by 0.64%, while using our GLA
module achieves higher accuracy (42.66% vs. 42.58%). Our GLA also improves the
performance of the model when both facial and context information is used to
predict emotion. Specifically, our model with GLA achieves the best result
with an accuracy of 46.91%, which is higher than the method with no attention
3.72% and standard attention module in caer  2.77%. The results from Table 4
show the effectiveness of our Global-Local Attention module for the task of
emotion recognition. They also verify that the use of both the local face
region and global context information is essential for improving emotion
recognition accuracy.

In order to emphasize the contribution of the Attention module to the final
results, we conduct Stuart-Maxwell test for each pair of methods that are used
in the setting (iv) of Table 4.The Stuart-Maxwell test is the generalized
version of McNemar test mcnemar_test  which is generally used for testing the
significant difference of multi-class classification models. The resulted
$p$-values of the tests are shown in Table 5. Note that the lower $p$ value
indicates stronger statistical disagreement between the two compared methods.
Overall, we can see that all of the models have significant different error
rates. Furthermore, the higher value on the main diagonal would imply stronger
agreement between the model prediction and the observed data, which means the
performance is better. In conjunction with the results in Table 4, we can
statistically confirm that our GLA module performs better than other attention
mechanisms.

####  4.3.4 Fusion methods comparison

Table 6: Results of different fusion strategies on the NCAER-S dataset. Methods | Accuracy (%)
---|---
Dubey et al. lbpdad  | 44.33
GLAMOR-Net + _Fusion Add_ | 45.62
GLAMOR-Net + _Fusion Max_ | 46.26
GLAMOR-Net + _Fusion Net_ | 46.91

To study the effectiveness of the information obtained from multiple
modalities via different fusion strategies, we conduct experiment by
alternatively changing the Fusion Module with multiple Fusion operators while
keeping other components of the system unchanged. Specifically, the Element-
wise Addition (_Fusion Add_), Element-wise Maximum (_Fusion Max)_) and our
_Fusion Net_ are studied in our experiment. Furthermore, we also compare our
method with recent work by Dubey et al. lbpdad . Table 6 summarizes the
results from our experiment. As shown in this table, the performance of our
network using _Fusion Net_ is superior to other fusion strategies. However, we
notice that the results from other fusion techniques are also very
competitive. This shows that the fusion strategy is also an important module
in the emotion recognition task, however the final result is also affected by
the extracted features from the feature extraction and attention modules.

####  4.3.5 Backbone architectures

Table 7: Accuracy of different encoding network architectures

Method | Backbone | #Params | CAER-S | NCAER-S
---|---|---|---|---
GLAMOR-Net | Original | 2.23M | 77.90 | 46.91
GLAMOR-Net | MobileNetV2 mobilenetv2  | 5.83M | 85.44 | 47.52
GLAMOR-Net | ResNet-18 resnet  | 22.90M | 89.88 | 48.40

We further study the effect of different Encoding network architectures.
Specifically, the MobileNetV2 mobilenetv2  and ResNet-18 resnet  are adopted
as the backbone network to extract features for both face and context branches
in our study. We use the output of the last convolutional layer as the
represented feature maps. These feature maps are then fed into the GLA module
and processed as in Section 3. We summarize the total amount of network
parameters and the classification results on CAER-S and NCAER-S in Table 7. We
observe that the ResNet-18 significantly outperforms other shallower
architectures (Original and MobileNetV2) and yields the best performance with
89.88% and 48.40% accuracy on CAER-S and NCAER-S. However, using such complex
model resulted in more memory footprint as well as computational cost.
Additionally, the MobileNetV2 can balance the trade-off between accuracy and
the speed of the model, which is a considerable option for deploying in
environments with limited resources such as mobile devices.

(i) |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
(ii) |  |  |  |  |  |  |
(iii) |  |  |  |  |  |  |
(iv) |  |  |  |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v) (a) Anger

(w) (b) Disgust

(x) (c) Fear

(y) (d) Happy

(z) (e) Neutral

(aa) (f) Sad

(ab) (g) Surprise

Figure 10: Predictions on the NCAER-S test set. The first two rows (i) and
(ii) show the results of the CAER-Net-S while the last two rows (iii) and (iv)
demonstrate predictions of our GLAMOR-Net. The columns’ names from (a) to (g)
denote the ground-truth emotion of the images

|  |
---|---|---
|  |

(a)

(b)

(c)

(d)

(e)

(f)

Figure 11: Human emotion detection results in the wild setting

####  4.3.6 Visualization

Fig. 9 shows the qualitative visualization with learned attention maps
obtained by our method GLAMOR-Net in comparison with CAER-Net-S. It can be
seen that our Global-Local attention mechanism produces better saliency maps
and helps the model attend to the right discriminative regions in the
surrounding background than the attention map produced by CAER-Net-S caer . As
we can see, our model is able to focus on the gesture of the person (Fig. 9f)
and also the face of surrounding people (Fig. 9c, Fig. 9d) to infer the
emotion accurately.

Fig. 10 shows some emotion recognition results of different approaches on the
NCAER-S dataset. More specifically, the first two rows (i) and (ii) contain
predictions of the CAER-Net-S while the last two rows (iii) and (iv) show the
results of our GLAMOR-Net. In some cases, our model was able to exploit the
context effectively to perform inference accurately. For instance, with the
same sad image input (shown on the (i) and (iii) rows), the CAER-Net-S
misclassified it as neutral while the GLAMOR-Net correctly recognized the true
emotion category. It might be because our model was able to identify that the
man was hugging and appeasing the woman and inferred that they were sad.
Another example is shown on the (i) and (iii) rows of the fear column. Our
model classified the input accurately, while the CAER-Net-S might be confused
between the facial expression and the wedding surrounding, thus incorrectly
predicted the emotion as happy.

On the other hand, we can also see on the (iv) rows of Fig. 10, the GLAMOR-Net
misclassified the disgust and the surprise images as happy and the neutral
image as sad. The reason might be that these images look quite confusing even
to humans. Our model also failed to recognize emotions in the anger, fear,
happy and sad images on the (iv) rows and predicted them as neutral instead.
It can be because the facial expression in these images does not manifest
clearly enough, which makes it difficult to distinguish between the neutral
class and these emotion categories. This uncertainty was previously shown in
the confusion matrix in Fig. 8.

####  4.3.7 Emotion Recognition in The Wild

As both the CAER-S dataset and its new split NCAER-S dataset contain only
images from movie settings, they have a very limited number of people in a
constrained environment. Therefore, the model trained using these datasets
potentially do not work well on real-world image setting. Despite this
challenge, Fig. 11 shows that our GLAMOR-Net can successfully detect and
recognize human emotion in these challenging settings. Note that, the input
images in this setup do not share any overlap with the movie settings as in
the training set. This again confirms the generalization ability of our
proposed method.

##  5 Conclusions and Future Work

In this work, we presented a novel method to exploit context information more
efficiently by using the proposed global-local attention model. We have shown
that our approach can considerably improve the emotion classification accuracy
compared to the current state-of-the-art result in the context-aware emotion
recognition task. The results on the CAER-S and the NCAER-S dataset
consistently demonstrate the effectiveness and robustness of our method.

Our approach currently only takes static images as input, which limits the
amount of knowledge that can be exploited. We are planning to utilize temporal
information in dynamic videos and other modalities such as audio in order to
further improve the performance. We also consider releasing a more challenging
emotion recognition dataset that contains rich background contexts with
multiple faces in the same frame and take advantage of our attention model to
extract the context saliency map for each face in a more effective manner. We
hope that our work will pave the way for future work in which predicting the
emotions of different people simultaneously is tackled.

## Declarations

Conflict of interest The authors declare that they have no conflict of
interest.

Availability of data and material The NCAER-S dataset can be downloaded at
https://bit.ly/NCAERS_dataset.

Code availability The source code and trained model of our network are
available at https://github.com/minhnhatvt/glamor-net.

## References

  * (1) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: Tensorflow: A system for large-scale machine learning.  In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI’16, p. 265–283 (2016)
  * (2) Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate.  In: Y. Bengio, Y. LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015).  URL http://arxiv.org/abs/1409.0473
  * (3) Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel.  In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, CIVR ’07, p. 401–408. Association for Computing Machinery, New York, NY, USA (2007).  DOI 10.1145/1282280.1282340.  URL https://doi.org/10.1145/1282280.1282340
  * (4) Castellano, G., Kessous, L., Caridakis, G.: Emotion recognition through multiple modalities: Face, body gesture, speech.  In: C. Peter, R. Beale (eds.) Affect and Emotion in Human-Computer Interaction, From Theory to Applications, _Lecture Notes in Computer Science_ , vol. 4868, pp. 92–103. Springer (2008).  DOI 10.1007/978-3-540-85099-1˙8.  URL https://doi.org/10.1007/978-3-540-85099-1_8
  * (5) Chen, J., Chen, Z., Chi, Z., Fu, H.: Facial expression recognition based on facial components detection and hog features (2014)
  * (6) Chen, Y., Wang, J., Chen, S., Shi, Z., Cai, J.: Facial motion prior networks for facial expression recognition.  In: 2019 IEEE Visual Communications and Image Processing, VCIP 2019, Sydney, Australia, December 1-4, 2019, pp. 1–4. IEEE (2019).  DOI 10.1109/VCIP47243.2019.8965826.  URL https://doi.org/10.1109/VCIP47243.2019.8965826
  * (7) Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition.  In: C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama, R. Garnett (eds.) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 577–585 (2015).  URL http://papers.nips.cc/paper/5847-attention-based-models-for-speech-recognition
  * (8) Clark, E.A., Kessinger, J., Duncan, S.E., Bell, M.A., Lahne, J., Gallagher, D.L., O’Keefe, S.F.: The facial action coding system for characterization of human affective response to consumer product-based stimuli: A systematic review.  Frontiers in Psychology 11, 920 (2020).  DOI 10.3389/fpsyg.2020.00920.  URL https://www.frontiersin.org/article/10.3389/fpsyg.2020.00920
  * (9) Clavel, C., Vasilescu, I., Devillers, L., Richard, G., Ehrette, T.: Fear-type emotion recognition for future audio-based surveillance systems.  Speech Communication 50, 487–503 (2008).  DOI 10.1016/j.specom.2008.03.012
  * (10) Corneanu, C.A., Simón, M.O., Cohn, J.F., Guerrero, S.E.: Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications.  IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8), 1548–1568 (2016)
  * (11) Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.: Emotion recognition in human-computer interaction.  Signal Processing Magazine, IEEE 18, 32 – 80 (2001).  DOI 10.1109/79.911197
  * (12) Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection.  In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Volume 1 - Volume 01, CVPR ’05, p. 886–893. IEEE Computer Society, USA (2005).  DOI 10.1109/CVPR.2005.177.  URL https://doi.org/10.1109/CVPR.2005.177
  * (13) Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Collecting large, richly annotated facial-expression databases from movies.  IEEE MultiMedia 19(3), 34–41 (2012).  DOI 10.1109/MMUL.2012.26
  * (14) Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms.  Neural Computation 10(7), 1895–1923 (1998).  DOI 10.1162/089976698300017197
  * (15) Do, T., Nguyen, B.X., Tjiputra, E., Tran, M., Tran, Q.D., Nguyen, A.: Multiple meta-model quantifying for medical visual question answering.  arXiv preprint arXiv:2105.08913 (2021)
  * (16) Do, T.T., Nguyen, A., Reid, I.: Affordancenet: An end-to-end deep learning approach for object affordance detection.  In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 5882–5889. IEEE (2018)
  * (17) Dubey, S.R., Roy, S.K., Chakraborty, S., Mukherjee, S., Chaudhuri, B.B.: Local bit-plane decoded convolutional neural network features for biomedical image retrieval.  Neural Computing and Applications 32(11), 7539–7551 (2020).  DOI 10.1007/s00521-019-04279-6.  URL https://doi.org/10.1007/s00521-019-04279-6
  * (18) Ekman, P., Friesen, W.: Constants across cultures in the face and emotion.  Journal of personality and social psychology 17(2), 124—129 (1971).  DOI 10.1037/h0030377.  URL https://doi.org/10.1037/h0030377
  * (19) El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classification schemes, and databases.  Pattern Recognition 44(3), 572 – 587 (2011).  DOI https://doi.org/10.1016/j.patcog.2010.09.020.  URL http://www.sciencedirect.com/science/article/pii/S0031320310004619
  * (20) Evgeniou, T., Pontil, M.: Support vector machines: Theory and applications.  In: Machine Learning and Its Applications, pp. 249–257. Springer Berlin Heidelberg (2001).  DOI 10.1007/3-540-44673-7˙12.  URL https://doi.org/10.1007/3-540-44673-7_12
  * (21) Galassi, A., Lippi, M., Torroni, P.: Attention in natural language processing.  IEEE Transactions on Neural Networks and Learning Systems (2020)
  * (22) Gao, Q., Zeng, H., Li, G., Tong, T.: Graph reasoning-based emotion recognition network.  IEEE Access 9, 6488–6497 (2021).  DOI 10.1109/ACCESS.2020.3048693
  * (23) Georgescu, M., Ionescu, R.T., Popescu, M.: Local learning with deep and handcrafted features for facial expression recognition.  IEEE Access 7, 64827–64836 (2019).  DOI 10.1109/ACCESS.2019.2917266
  * (24) Han, K., Yu, D., Tashev, I.: Speech emotion recognition using deep neural network and extreme learning machine (2014)
  * (25) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
  * (26) Hyun, K.H., Kim, E.H., Kwak, Y.K.: Emotion Recognition Using Voice Based on Emotion-Sensitive Frequency Ranges, pp. 217–223.  Springer Berlin Heidelberg, Berlin, Heidelberg (2007).  DOI 10.1007/978-3-540-73424-6˙25.  URL https://doi.org/10.1007/978-3-540-73424-6_25
  * (27) Jordan, M.I.: Graphical models.  Statistical Science 19(1), 140–155 (2004).  DOI 10.1214/088342304000000026.  URL https://doi.org/10.1214/088342304000000026
  * (28) King, D.: Dlib-ml: A machine learning toolkit.  J. Mach. Learn. Res. 10, 1755–1758 (2009)
  * (29) Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Context based emotion recognition using emotic dataset.  IEEE Transactions on Pattern Analysis and Machine Intelligence PP, 1–1 (2019).  DOI 10.1109/TPAMI.2019.2916866
  * (30) Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks.  In: F. Pereira, C.J.C. Burges, L. Bottou, K.Q. Weinberger (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc. (2012).  URL https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * (31) Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks.  In: Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, p. 1097–1105. Curran Associates Inc., Red Hook, NY, USA (2012)
  * (32) Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks.  In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10142–10151 (2019).  DOI 10.1109/ICCV.2019.01024
  * (33) Lee, J.B., Rossi, R.A., Kim, S., Ahmed, N.K., Koh, E.: Attention models in graphs: A survey.  ACM Trans. Knowl. Discov. Data 13(6) (2019).  DOI 10.1145/3363574.  URL https://doi.org/10.1145/3363574
  * (34) Li, S., Deng, W.: Deep facial expression recognition: A survey.  IEEE Transactions on Affective Computing p. 1–1 (2020).  DOI 10.1109/taffc.2020.2981446.  URL http://dx.doi.org/10.1109/TAFFC.2020.2981446
  * (35) Liu, X., Kumar, B.V.K.V., You, J., Jia, P.: Adaptive deep metric learning for identity-aware facial expression recognition.  In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 522–531 (2017)
  * (36) Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010).  DOI 10.1109/CVPRW.2010.5543262
  * (37) Matsumoto, D.: More evidence for the universality of a contempt expression.  Motivation and Emotion 16, 363–368 (1992)
  * (38) Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos.  In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 3866–3870 (2019).  DOI 10.1109/ICIP.2019.8803603
  * (39) Mittal, T., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emoticon: Context-aware multimodal emotion recognition using frege’s principle.  In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14222–14231 (2020).  DOI 10.1109/CVPR42600.2020.01424
  * (40) Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing 10(1), 18–31 (2019).  DOI 10.1109/TAFFC.2017.2740923
  * (41) Nguyen, A., Do, T.T., Reid, I., Caldwell, D.G., Tsagarakis, N.G.: V2cnet: A deep learning framework to translate videos to commands for robotic manipulation.  arXiv preprint arXiv:1903.10869 (2019)
  * (42) Nguyen, A., Nguyen, N., Tran, K., Tjiputra, E., Tran, Q.D.: Autonomous navigation in complex environments with deep multimodal fusion network.  In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5824–5830. IEEE (2020)
  * (43) Nguyen, B.X., Nguyen, B.D., Do, T., Tjiputra, E., Tran, Q.D., Nguyen, A.: Graph-based person signature for person re-identifications.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 3492–3501 (2021)
  * (44) Oliva, A., Torralba, A.: Torralba, a.: Building the gist of a scene: The role of global image features in recognition. progress in brain research 155, 23-36.  Progress in brain research 155, 23–36 (2006).  DOI 10.1016/S0079-6123(06)55002-2
  * (45) Päivärinta, J., Rahtu, E., Heikkilä, J.: Volume local phase quantization for blur-insensitive dynamic texture classification.  In: A. Heyden, F. Kahl (eds.) Image Analysis, pp. 360–369. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)
  * (46) Paulmann, S., Bleichner, M., Kotz, S.A.: Valence, arousal, and task effects in emotional prosody processing.  Frontiers in Psychology 4 (2013).  DOI 10.3389/fpsyg.2013.00345.  URL https://doi.org/10.3389/fpsyg.2013.00345
  * (47) Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A., Manocha, D.: Identifying emotions from walking using affective and deep features (2020)
  * (48) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks (2019)
  * (49) Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: A survey of registration, representation, and recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence (2015)
  * (50) Schindler, K., Van Gool, L., de Gelder, B.: Recognizing emotions expressed by body pose: A biologically inspired neural model.  Neural Networks 21(9), 1238 – 1246 (2008).  DOI https://doi.org/10.1016/j.neunet.2008.05.003.  URL http://www.sciencedirect.com/science/article/pii/S0893608008000944
  * (51) Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: A comprehensive study.  Image and Vision Computing 27(6), 803 – 816 (2009).  DOI https://doi.org/10.1016/j.imavis.2008.08.005.  URL http://www.sciencedirect.com/science/article/pii/S0262885608001844
  * (52) Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G.: Multiple kernel learning for emotion recognition in the wild.  ICMI 2013 - Proceedings of the 2013 ACM International Conference on Multimodal Interaction (2013).  DOI 10.1145/2522848.2531741
  * (53) Sikka, K., Wu, T., Susskind, J., Bartlett, M.: Exploring bag of words architectures in the facial expression domain.  In: Proceedings of the 12th International Conference on Computer Vision - Volume 2, ECCV’12, p. 250–259. Springer-Verlag, Berlin, Heidelberg (2012).  DOI 10.1007/978-3-642-33868-7˙25.  URL https://doi.org/10.1007/978-3-642-33868-7_25
  * (54) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  In: Y. Bengio, Y. LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015).  URL http://arxiv.org/abs/1409.1556
  * (55) Stathopoulou, I.O., Tsihrintzis, G.A.: Emotion recognition from body movements and gestures.  In: G.A. Tsihrintzis, M. Virvou, L.C. Jain, R.J. Howlett (eds.) Intelligent Interactive Multimedia Systems and Services, pp. 295–303. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)
  * (56) Sun, B., Li, L., Zhou, G., Wu, X., He, J., Yu, L., Li, D., Wei, Q.: Combining multimodal features within a fusion network for emotion recognition in the wild.  In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI ’15, p. 497–502. Association for Computing Machinery, New York, NY, USA (2015).  DOI 10.1145/2818346.2830586.  URL https://doi.org/10.1145/2818346.2830586
  * (57) Wang, F., Tax, D.M.J.: Survey on the attention based RNN model and its applications in computer vision.  CoRR abs/1601.06823 (2016).  URL http://arxiv.org/abs/1601.06823
  * (58) Wang, K., Peng, X., Yang, J., Meng, D., Qiao, Y.: Region attention networks for pose and occlusion robust facial expression recognition.  CoRR abs/1905.04075 (2019).  URL http://arxiv.org/abs/1905.04075
  * (59) Zhao, Z., Liu, Q., Wang, S.: Learning deep global multi-scale and local attention features for facial expression recognition in the wild.  IEEE Transactions on Image Processing 30, 6544–6556 (2021)
  * (60) Zhao, Z., Liu, Q., Zhou, F.: Robust lightweight facial expression recognition network with label distribution training.  In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3510–3519 (2021)


# Signatures of generalized ALP interactions in SM decays of mesons

Subhajit Ghosh111Speaker Department of Physics and Astronomy, University of
Notre Dame, South Bend, IN 46556, USA  Triparno Bandyopadhyay  Tuhin S. Roy
Department of Theoretical Physics, Tata Institute of Fundamental Research,
Mumbai 400005, India.

###### Abstract

In addition to giving rise to spectacular new physics signals in the final
states of meson decays, Axion-like-particles also induce modifications in the
standard model decays of mesons. These ‘indirect’ signatures can be
parametrized as the modifications of the hadronic form factors and can be
probed using meson decay width and decay distribution measurements. Starting
with a generalized ALP Lagrangian, we demonstrate these effects for
semileptonic Kaon decays and derived bounds using NA48/2 data. We also briefly
discuss other indirect signatures such as modification of meson mass spectrum
and ‘sum rules’ comprised of meson decay amplitudes which show deviation in
presence of ALP.

##  I Introduction

Proposed as a dynamical solution to the Strong CP problem Peccei and Quinn
(1977), Axion has emerged as a highly versatile tool for new physics model
building (for review, see Ref. Choi et al. (2021); Bauer et al. (2017, 2021)).
Immense experimental efforts are ongoing to detect Axion or Axion-like-
particle (ALP) through its interaction with quarks, leptons, and the gauge
bosons. The majority of these experiments belong to the class of ‘direct’
searches, where the ALP is either present in the initial state or is
subsequently produced via the interaction of Standard Model particles. In the
latter case, the final state ALP leaves its signature in the detector either
decaying back into Standard Model (SM) particles or in the form of missing
energy if its decay lifetime is beyond the experimental sensitivity.

Interestingly, there exists a completely independent way of searching for
these particles by looking into the decays of mesons in SM final states. These
‘indirect’ searches do not contain the ALP in either the initial or the final
state of the decay process. ALP, in general, has mass mixing and kinetic
mixing interactions with the neutral mesons. Diagonalization of these mixings
in the physical basis modifies the decay rates of mesons to the SM states.
Thus deviations of these decay rates from the SM predictions can put
independent constraints on the ALP couplings with the SM. Since the ALP does
not take part in these decay processes, the bounds derived from indirect
searches are independent of ALP lifetime or ALP decay channels.

In Ref. Bandyopadhyay et al. (2022), we demonstrated how precision
measurements of semileptonic decays of charged Kaon can be used to put
constraints on the ALP parameter space. We further showed how these indirect
searches complement the direct searches of ALP. We also construct ‘sum rules’
using partial decay widths of mesons which can distinguish the effects of
different classes of ALP operators.

##  II Generalized ALP Lagrangian

In Ref. Bandyopadhyay et al. (2022), we start out with a ‘generalized’ set of
operators upto dimension $d=5$ in the electroweak basis involving the ALP and
the quarks, where only the periodic symmetry of the ALP is taken into account.
These operators are

$\sum_{i=0}^{8}\left(C_{L}^{i}\mathcal{O}_{L}^{i}+C_{R}^{i}\mathcal{O}_{R}^{i}+C_{LR}^{i}\mathcal{O}_{LR}^{i}\right)+C_{W}\mathcal{O}_{W}+C_{Z}\mathcal{O}_{Z}\;,$
(1)

with $C_{a}$s being the _real_ Wilson coefficients for the operator of class
$\mathcal{O}_{a}$. We tabulate the operators in Table. 1.

$\mathcal{O}_{L}^{i}$ | $\dfrac{1}{f_{a}}\partial_{\mu}a\>\overline{q}_{L}t^{i}\gamma^{\mu}q_{L}$ |  $i=0,8:$ Allowed
---|---|---
$i=1,2,4,5:$ break EM
$i=1(1)7:$ break $SU(2)_{W}$
$i=6,7:$ tree FCNC
$\mathcal{O}_{R}^{i}$ |  $\dfrac{1}{f_{a}}\partial_{\mu}a\>\overline{q}_{R}t^{i}\gamma^{\mu}q_{R}$ |
$i=0,3,8:$ Allowed
|  |  $i=1,2,4,5:$ break EM
$\mathcal{O}_{LR}^{i}$ |  $\dfrac{a}{f_{a}}\>\overline{q}_{L}t^{i}Mq_{R}$ |  $i=6,7:$ tree FCNC
$\mathcal{O}_{W}$ | $-\dfrac{a}{f_{a}}\>\overline{q}_{L}Q^{W}\cancel{j}_{\pm}q_{L}$ |
$\mathcal{O}_{Z}$ |  9 cm $-\dfrac{a}{f_{a}}\>\left(\overline{q}_{L}Q_{L}^{Z}\;\cancel{j}_{Z}q_{L}\right.$ |
$\left.+\ \overline{q}_{R}Q_{R}^{Z}\;\cancel{j}_{Z}q_{R}\right)$ |  |
Table 1: The dimension-5 ALP operators in the EW basis. The rightmost column
indicates the generators allowed by symmetry and also the ones not considered
by us. $t^{i}$s are the $SU(3)$ generators and $t^{0}$ is the identity matrix.
The $j^{\mu}_{\pm}$ and $j^{\mu}_{Z}$ are current replacements of
$W_{\pm}^{\mu}$ and $Z^{\mu}$, respectively.

This framework includes operators
$\left(\mathcal{O}_{W},\mathcal{O}_{Z}\right)$ where the ALP appears as a
scalar and not a pseudo-scalar, allowing us to include models where the ALP
may mix with CP even states. In this study, we have only considered operators
linear in $a/f_{a}$ where $f_{a}$ is the ALP decay constant.

We match the quark Lagrangian in Eq. (1) with the meson Lagrangian using
leading order $SU(3)$ $\chi$EFT. Operator $\mathcal{O}_{LR}^{i}$ induces mass
mixing between the neutral mesons and the ALP, whereas, $\mathcal{O}_{L}^{i}$
and $\mathcal{O}_{R}^{i}$ generate kinetic mixing. The Lagrangian, after basis
diagonalization, captures all the essential physics of indirect signatures of
the ALP.

###  II.1 Modification of the form factors for $K_{\ell 3}$ decays

Due to the mixing of the ALP with the neutral mesons the form factors (FF) of
the mesonic states get modified. In Ref. Bandyopadhyay et al. (2022) we
focussed on the $K^{+}\to\pi^{0}\ell^{+}\nu$ ($K^{+}_{\ell_{3}}$) decays and
studied the effects of ALP interaction on strangeness violating FF defined as:

$\langle\pi^{0}(p_{\pi})|\bar{s}\gamma_{\mu}u|K^{+}(p_{K})\rangle\\\
\equiv\frac{1}{\sqrt{2}}\Big{[}f_{+}^{K^{+}\pi^{0}}(q^{2})\;Q_{\mu}+f_{-}^{K^{+}\pi^{0}}(q^{2})\;q_{\mu}\Big{]},\\\
\mathrm{where,}\quad Q^{\mu}=p_{K}^{\mu}+p_{\pi}^{\mu};\quad
q_{\mu}=p_{K}^{\mu}-p_{\pi}^{\mu}.$ (2a)

At leading order $\left(\mathcal{O}(p^{2})\right)$ $\chi$EFT, the amplitude of
$K_{\ell_{3}}$ process in terms of the FF is written as;

$\mathcal{A}=G_{F}V_{\bar{s}u}\left[\tilde{f}_{+}^{K^{+}\pi^{0}}(0)\,Q_{\mu}+\tilde{f}_{-}^{K^{+}\pi^{0}}(0)\,q_{\mu}\right]\\\
\times\bar{u}_{\nu}\gamma^{\mu}\frac{1}{2}\left(1-\gamma_{5}\right)v_{\ell}.$
(3)

Here $G_{F}$ is the Fermi constant, $V_{\bar{s}u}$ is the CKM matrix elements
and $\tilde{f}_{\pm}^{K^{+}\pi^{0}}(0)$ are the ‘effective’ FF which include
additional contributions for the hadronic amplitude beyond the modifications
of $f_{\pm}^{K^{+}\pi^{0}}$ Bandyopadhyay et al. (2022). The ALP-induced
modifications of the FF are parametrized as;

$\displaystyle\begin{split}&\tilde{f}_{+}^{K^{+}\pi^{0}}(0)=\alpha^{(0)}_{K^{+}\pi_{0}}+\xi^{2}\left(\alpha^{(2)}_{K^{+}\pi_{0}}+i\tilde{\alpha}^{(2)}_{K^{+}\pi_{0}}\right)\;,\\\
&\tilde{f}_{-}^{K^{+}\pi^{0}}(0)=\beta^{(0)}_{K^{+}\pi_{0}}+\xi^{2}\left(\beta^{(2)}_{K^{+}\pi_{0}}+i\tilde{\beta}^{(2)}_{K^{+}\pi_{0}}\right),\text{with}~{}~{}\\\
&\alpha^{(0)}_{K^{+}\pi_{0}}=1-\sqrt{3}\,\epsilon,\\\
&\alpha^{(2)}_{K^{+}\pi_{0}}=-\frac{C_{3}}{8}(C_{LR}^{3}-C_{R}^{3}+2\sqrt{3}(C_{LR}^{8}-C_{R}^{8})),\\\
&\tilde{\alpha}^{(2)}_{K^{+}\pi_{0}}=-\frac{1}{2}C_{3}C_{W},\
\text{and}~{}~{}\\\
&\beta^{(0)}_{K^{+}\pi_{0}}=0,\;\beta^{(2)}_{K^{+}\pi_{0}}=-\frac{\sqrt{3}}{4}C_{3}C_{L}^{8},\;\tilde{\beta}^{(2)}_{K^{+}\pi_{0}}=\frac{1}{2}C_{3}C_{W},\\\
&\text{where}~{}\epsilon={\sqrt{3}\over 4}{m_{u}-m_{d}\over
m_{s}-(m_{u}+m_{d})/2},\\\
&\text{and}~{}C_{3}=C_{LR}^{3}-C_{R}^{3}.\end{split}$ (4)

Figure 1:  Bottom: Differential spectrum of $K^{+}_{\mu_{3}}$, w.r.t.
$E_{\pi}$ for $\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}=0.05$ (dashed red) and
$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}=0.51$ (dash-dotted blue). The values of the
parameters are chosen such that the total decay width is same for both graphs.
The SM spectrum is shown in black.

Here $\alpha^{(0)}$ and $\beta^{(0)}$ correspond to the SM contributions at
leading order in $\chi$EFT. The imaginary pieces of the ALP-induced
modifications of the FF contribute at $\mathcal{O}(\xi^{4})$, whereas, the
real parts, which interfere with the SM contribution, produce effects at
$\mathcal{O}(\xi^{2})$. Therefore, we drop the imaginary contributions from
our calculation. We incorporate the higher order $\chi$EFT effects in this
formalism, which introduce momentum dependence in the FF, via

$\displaystyle\mathrm{Re}\left(\tilde{f}_{+}^{K^{+}\pi^{0}}(t)\right)$
$\displaystyle\simeq\left[1+\xi^{2}\frac{\alpha^{(2)}_{K^{+}\pi^{0}}}{\alpha^{(0)}_{K^{+}\pi^{0}}}\\!\right]f_{+,\,\mathrm{SM}}^{K^{+}\pi^{0}}(t)\;,$
(5) $\displaystyle\mathrm{Re}\left(\tilde{f}_{-}^{K^{+}\pi^{0}}(t)\right)$
$\displaystyle\simeq\left[1+\xi^{2}\frac{\beta^{(2)}_{K^{+}\pi^{0}}}{\delta\beta^{(0)}_{K^{+}\pi^{0}}}\\!\right]f_{-,\,\mathrm{SM}}^{K^{+}\pi^{0}}(t)\;.$
(6)

Here $\delta\beta^{(0)}_{K^{+}\pi^{0}}$ is the higher order $\chi$EFT
contribution to $\beta^{(0)}_{K^{+}\pi^{0}}$. In writing the above equations,
we have ignored the ALP-induced corrections to the ‘slope parameters’, which
are subleading in nature.

Figure 2: Bottom Left: Binned Dalitz plot for the residual (the difference
between the data and the SM expectation). Top Right: Expected Dalitz plot
signal for $\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}=0.06$. The bottom-right and top-
left are the marginalized residuals (with errorbar) and signals with respect
to $E_{\mu}$ and $E_{\pi}$, respectively.

Fig. 1 shows the modification of the $K_{\ell_{3}}$ differential rate.
$\xi^{2}\alpha^{(2)}$, which modifies the dominant
$\tilde{f}_{+}^{K^{+}\pi^{0}}$ FF, results in almost constant modification
with respect to SM. Whereas $\xi^{2}\beta^{(2)}$ induces signification
distortion of the differential distribution. Note that, for the same values of
the ALP parameters, the modification due to $\xi^{2}\beta^{(2)}$ is much
smaller, since its effect is suppressed by the final state lepton mass.

##  III Constraints

In this section, we depict the current constraints of the ALP parameters from
$K_{\ell_{3}}$ decays derived using the following datasets:

  * •

The decay distribution measurements for $K_{\mu_{3}}$ and $K_{e_{3}}$ by the
NA48/2 collaboration Batley et al. (2018),

  * •

Measured total decay widths of the above two decays Zyla et al. (2020).

We compare the theoretical SM expectations of the decay rates and the decay
distributions with the above experimental datasets. For the theoretical SM
calculation, we use the FF parameters obtained from lattice computations by
the European twisted mass collaboration Carrasco et al. (2016). We include the
uncertainties of and the correlations among the theoretical values of the FF
parameters in our computation Bandyopadhyay et al. (2022); Carrasco et al.
(2016).

In Fig. 2 we compare the theoretical expectation with the measured data for
$K_{\mu 3}$ decay. The figure shows the residual Dalitz plot distribution
which is the difference between the data and the theoretical calculation. We
also show the 1D residual plots projected along $E_{\mu}$ and $E_{\pi}$
containing the experimental errorbar. The 1D residual with respect to
$E_{\mu}$ shows a slight systematic excess which can be fitted using
$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}=0.06$. However, the excess is not
significant and becomes consistent with the SM when the theoretical
uncertainties of the FF are taken into account.

Figure 3: The allowed regions for the ALP parameters in the $\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}$ \- $\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ plane at 95% C.L. The black contour shows the constrains from the $K_{\mu 3}$ distribution measurement. The yellow and the hached regin are the constrains derived from the total decay rate of $K_{e3}$ and $K_{\mu 3}$, respectively. The red region shows the combined constrain. The white cross denotes the SM point.  |  12 cm$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ |
---|---|---
$\left(\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}=0\right)$ |  12 cm$\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}$ |
$\left(\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}=0\right)$ |  |
$K_{\mu_{3}}^{+}$ | $[-0.006,0.026]$ | $[-0.021,0.007]$
$K_{\mu_{3}}^{+}+K_{e_{3}}^{+}$ | $[-0.006,0.026]$ | $[-0.018,0.003]$
Table 2: The 95% confidence limits on $\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}$ and
$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ from the $K_{\mu_{3}}^{+}$ decay analysis
and the combined $K_{\mu_{3}}^{+}+K_{e_{3}}^{+}$ analysis.

We performed a $\chi^{2}$ analysis using all the datasets quoted at the
beginning of the section. The contours at 95% C.L. in the
$\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}$ \- $\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$
plane are shown in Fig. 3. The limits at 95% C.L. from the analysis are shown
in Table. 2. The constraints on $\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ from the
total width measurements of both $K_{\mu_{3}}$ and $K_{e_{3}}$ are
substantially weaker owing to the associated lepton mass suppression. However,
from the decay distribution measurements, the constraint on
$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ is much stronger compared to
$\xi^{2}\alpha^{(2)}_{K^{+}\pi^{0}}$. This is due to the fact that the decay
distribution measurement of NA48/2 is mostly sensitive to the shape
modification of the spectrum, not the normalization ( i.e, not sensitive to
the overall scaling of total decay rate). Since,
$\xi^{2}\beta^{(2)}_{K^{+}\pi^{0}}$ induces a dissimilar momentum dependence
with respect to the SM (as shown in Fig. 3), the constraint is stricter in
this case Bandyopadhyay et al. (2022). Note that, $K_{e_{3}}$ distribution
measurement does not provide any meaningful constraint on the ALP parameter
space due to the electron mass suppression. The combined constraint from both
decay width and distribution measurements is shown in red and is consistent
with the SM at 95% C.L.

##  IV Other indirect signatures

In addition to the FF modifications, ALP also induces a number of interesting
indirect signatures. Due to the mass and kinetic mixing with the ALP, the mass
spectra of the mesons deviate from the SM values. These deviations can be
constrained using theoretically clean mass ratio observables, such as the
well-known Gell-Mann-Okubo mass relation and the charged and neutral pion mass
difference ratio. In Ref. Bandyopadhyay et al. (2022) we sketch the resulting
constraints from these observables.

The constraints derived using the indirect method largely complement the ones
derived via direct observations. Using the generalized Lagrangian in Eq. 1, it
can be shown that for certain values of the ALP parameters the direct decay
amplitude for $K^{+}\to a\ell^{+}\nu$ goes to zero. Although the indirect
signature still persists in those limits, i.e, the amplitude for
$K^{+}\to\pi^{0}\ell^{+}\nu$ shows deviation from the SM expectation
Bandyopadhyay et al. (2022). Thus, indirect methods can shine light on the
direct detection blind spots generally known as ‘pion-phobia’.

We can also construct ‘sum-rules’ using SM decay widths of mesons which
provide a unique way of identifying the presence and nature of ALP
interactions in meson Lagrangian. As shown in Ref. Bandyopadhyay et al. (2022)
the deviations from these sums from unity point towards the presence of ALP
coupling with mesons, whereas, the signs of the deviations can be associated
with different classes of ALP operators. Therefore, these sums, which are
constructed out of low-energy observables, can shed light on the UV nature of
the ALP physics.

##  V Conclusion

Indirect searches open up a hitherto unexplored avenue of ALP phenomenology.
The presence of a large number of indirect channels and the complementarity
with the direct searches make this a viable approach to looking for ALP
signatures. With the improvement in the theoretical predictions of the FF and
high precision data from low-energy experiments, these bounds will improve
significantly in near future. Similar analysis involving other indirect
channels, such as heavy meson decays and tau decays, will be able to put
further constrain on the ALP parameter space.

###### Acknowledgements.

The authors thank Dmitry Madigozhin for providing them with the source of the
NA48/2 dataset. The research of SG is supported by the NSF grant PHY-2014165.

## References

  * Peccei and Quinn (1977) R. D. Peccei and H. R. Quinn, Phys. Rev. Lett. 38, 1440 (1977).
  * Choi et al. (2021) K. Choi, S. H. Im, and C. Sub Shin, Ann. Rev. Nucl. Part. Sci. 71, 225 (2021), eprint 2012.05029.
  * Bauer et al. (2017) M. Bauer, M. Neubert, and A. Thamm, JHEP 12, 044 (2017), eprint 1708.00443.
  * Bauer et al. (2021) M. Bauer, M. Neubert, S. Renner, M. Schnubel, and A. Thamm (2021), eprint 2110.10698.
  * Bandyopadhyay et al. (2022) T. Bandyopadhyay, S. Ghosh, and T. S. Roy, Phys. Rev. D 105, 115039 (2022), eprint 2112.13147.
  * Batley et al. (2018) J. R. Batley et al. (NA48/2), JHEP 10, 150 (2018), eprint 1808.09041.
  * Zyla et al. (2020) P. Zyla et al. (Particle Data Group), PTEP 2020, 083C01 (2020).
  * Carrasco et al. (2016) N. Carrasco, P. Lami, V. Lubicz, L. Riggio, S. Simula, and C. Tarantino, Phys. Rev. D 93, 114512 (2016), eprint 1602.04113.