11institutetext: School of Physics and Microelectronics, Zhengzhou University,
Zhengzhou, 450001, China22institutetext: Department of Physics, University of
Washington at Seattle,
Seattle, Washington 98195, USA33institutetext: Department of Physics and
Institute of Theoretical Physics,
Nanjing Normal University, Nanjing, 210023, China

# Detecting highly collimated photon-jets from Higgs boson exotic decays with
deep learning

Xiaocong Ai 2 William Y. Feng 2 Shih-Chieh Hsu 2 Ke Li 3 Chih-Ting Lu
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, there has been a growing focus on the search for anomalous objects
beyond standard model (BSM) signatures at the Large Hadron Collider (LHC).
This study investigates novel signatures involving highly collimated photons,
referred to as photon-jets. These photon-jets can be generated from highly
boosted BSM particles that decay into two or more collimated photons in the
final state. Since these photons cannot be isolated from each other, they are
treated as a single jet-like object rather than a multi-photon signature. The
Higgs portal model is utilized as a prototype for studying photon-jet
signatures. Specifically, GEANT4 is employed to simulate electromagnetic
showers in an ATLAS-like electromagnetic calorimeter, and three machine
learning techniques: Boosted Decision Trees (BDT), Convolutional Neural
Networks (CNN), and Particle Flow Networks (PFN) are applied to effectively
distinguish these photon-jet signatures from single photons and neutral pions
within the SM backgrounds. Our models attain an identification efficiency
exceeding $99\%$ for photon-jets, coupled with a rejection rate surpassing
$99\%$ for SM backgrounds. Furthermore, the sensitivities for searching
photon-jet signatures from the Higgs boson exotic decays at the High-
Luminosity LHC are obtained.

##  1 Introduction

The Higgs boson has held a position of central importance for over a decade
since its discovery CMS:2022dwd ; ATLAS:2022vkf . While an increasing body of
precise measurements has lent substantial support to its compatibility with
the Standard Model (SM) predictions, a fundamental question still looms large:
Is this particle indeed the SM Higgs boson, or might it be a SM-like
counterpart, affiliated with models beyond the SM? Such inquiries have led to
the exploration of Higgs portal models, representing a bridge between the SM
sector and the dark sector Arcadi:2019lka . Higgs portal models can address
diverse and pressing issues including neutrino mass and mixing
DeRomeri:2020wng , the nature of dark matter (DM) Arcadi:2019lka ;
Matsumoto:2018acr ; Bondarenko:2019vrb , the electroweak phase transition
Cohen:2012zza ; Huang:2012wn ; Chao:2014ina , cosmic inflation Bezrukov:2009yw
; Bezrukov:2021mio , and the hierarchy problem Batell:2022pzc . In the quest
to unravel the mysteries of these Higgs portal models, significant efforts are
made not only in precision measurements of Higgs boson couplings to SM
particles, but also in exploring its invisible and exotic decay channels
Shrock:1982kd ; Curtin:2013fra , which provide crucial insights into
previously uncharted territories.

As investigations at the Large Hadron Collider (LHC) have continued, the
absence of concrete evidence for new physics beyond the SM has left the
scientific community pondering a crucial possibility: perhaps conventional
search strategies, which have guided our exploration of new physics at the LHC
thus far, represent merely a subset of the manifold ways to unveil the enigmas
of new physics. New physics signal signatures may prove to be more intricate
and unconventional than previously imagined. In response to this evolving
landscape, the pursuit of anomalous objects beyond the SM has garnered growing
interest within the scientific community Chakraborty:2017mbz ; Alimena:2019zri
; Albouy:2022cin ; Franceschini:2022vck . Many of these searches for anomalous
objects defy the traditional methodologies used in the past, necessitating the
development of innovative search strategies and the deployment of cutting-edge
techniques. This shift in focus represents a profound and exciting frontier in
the ongoing journey of unraveling the mysteries of new physics.

Within the exploration of these anomalous objects, our focus centers on a
unique class of signal signatures: photon-jets Dobrescu:2000jt ; Toro:2012sv .
Photon-jets emerge as the byproducts of the decay of new, highly boosted light
particles, resulting in the generation of two or more closely collimated
photons in their final state. Their peculiarity lies in the fact that these
constituent photons cannot pass the conventional photon isolation criteria,
leading us to treat them as a jet-like object, instead of the multi-photon
signature Steinberg:2021iay ; Alves:2021puo ; Knapen:2021elo . The jet
substructure analysis has been applied to this novel photon-jet signature
Ellis:2012sd ; Ellis:2012zp ; Chakraborty:2017mbz ; Wang:2021uyb . On the
other hand, both ATLAS and CMS Collaborations have sought photon-jet
signatures using the traditional cut-and-count method and Boosted Decision
Trees (BDT) techniques ATLAS:2012soa ; Aaboud:2018djx ; CMS:2022wpu ;
ATLAS:2023eet . However, as the structure within the photon-jet becomes more
intricate, deep learning techniques have proven more powerful than previous
analysis strategies.

While previous studies have explored the application of Convolutional Neural
Networks (CNN) in the context of a simple scenario involving only two closely
collimated photons within the photon-jet signature Ren:2021prq ; Wang:2023pqx
, our ambition extends further. We aim to delve into the realm of more
intricate photon-jet structures, specifically exploring configurations with
two, four, and six closely collimated photons. This endeavor is undertaken
with the Higgs portal model as our concrete prototype, allowing us to explore
photon-jet signatures within this framework. In particular, we consider the
exotic decays of the Higgs boson into a pair of light scalars (dark Higgs
bosons) Toro:2012sv ; Ellis:2012sd ; Ellis:2012zp ; Chang:2015sdy ;
Sheff:2020jyw  or light pseudoscalars (axion-like particles) Dobrescu:2000jt ;
Toro:2012sv ; Draper:2012xt ; Aparicio:2016iwr ; Ellwanger:2016qax ;
Domingo:2016unq ; Chiang:2016eav ; Domingo:2016yih ; Wang:2021uyb ;
Ren:2021prq ; Lu:2022zbe ; Lane:2023eno . There is a substantial mass
difference between the scalar/pseudoscalar and the Higgs boson, leading to the
production of these light particles with high momentum and, consequently, the
formation of photon-jets when they predominantly decay into multiple photons
in the final state. To effectively distinguish these photon-jet signatures
from single photons and neutral pions within the SM backgrounds, we employ
three machine learning techniques: BDT Roe:2004na , CNN Ayyar:2020ijy , and
Particle Flow Network (PFN) Komiske:2018cqr . This approach extends beyond
conventional boundaries and offers a complementary method to new light
particle searches conducted at B-factory and fixed target experiments
Filimonova:2019tuy ; Belle-II:2020jti ; Ferber:2022rsf ; Belle-II:2023ueh ;
Blumlein:1990ay ; Dobrich:2015jyk ; Feng:2018pew ; Harland-Lang:2019zur ;
Gorbunov:2021ccu ; Kling:2022uzy ; Liu:2023bby .

The structure of this paper is organized as follows. In Sec. 2, we provide a
concise review of the Higgs portal model, focusing on the photon-jet
signatures within this framework. The application of three distinct machine
learning techniques to the identification of photon-jets and a demonstration
of their respective performances are presented in Sec. 3. We discuss the
obtained results and their implications for the Higgs portal model in Sec. 4.
Finally, a summary of our findings is presented in Sec. 5.

##  2 The Higgs portal model

In this study, we investigate a simplified Higgs portal model that extends
beyond the SM. This model introduces a framework in which, alongside the SM-
like Higgs boson denoted as $h_{1}$ with $m_{h_{1}}\simeq 125$ GeV, we
consider the presence of either a light scalar, $h_{2}$, or a light
pseudoscalar, $a$. These additional particles, namely $h_{2}$ and $a$, play
several roles within the model. The light scalar can potentially serve as a
mediator for sub-GeV DM models Matsumoto:2018acr ; Bondarenko:2019vrb , a
candidate for the light inflaton Bezrukov:2009yw ; Bezrukov:2021mio .
Moreover, a singlet complex scalar field is often used to provide the mass of
dark photon for dark sector models with extra $U(1)$ gauge symmetry, employing
the dark Higgs mechanism Baek:2014kna . Similarly, the light pseudoscalar can
manifest as an axion-like particle, serving as a mediator for sub-GeV DM
models Dolan:2014ska ; Dolan:2017osp ; Bharucha:2022lty ; Ghosh:2023tyz , and
even offering a solution to experimental anomalies, such as the muon $g-2$
excess Liu:2022tqn  and the MiniBooNE excess Chang:2021myh , among others.

In this framework, the spin-$0$ particles ($h_{1}$, $h_{2}$, and $a$) can
interact with one another or themselves via three-point and four-point
interactions. Our particular focus in this investigation centers on exploring
the $h_{1}$-$h_{2}$-$h_{2}$ and $h_{1}$-$a$-$a$ interactions, characterized by
the couplings $\mu_{h_{1}h_{2}h_{2}}$ and $\mu_{h_{1}aa}$, which have the same
dimension as mass. Based on the motivations outlined above, encompassing light
scalar and pseduoscalar, and with the aim of studying novel photon-jet
signatures, we concentrate on the sub-GeV scale $h_{2}$ and $a$ in this study
Bauer:2018uxu ; Winkler:2018qyg . Moreover, we assume that $h_{2}\rightarrow
aa$ is kinematically forbidden for the sake of simplicity. Concretely, the
partial decay width for $h_{1}\to h_{2}h_{2}$ can be represented as

$\Gamma(h_{1}\to h_{2}h_{2})=\frac{\mu^{2}_{h_{1}h_{2}h_{2}}}{32\pi
m_{h_{1}}}\sqrt{1-4\left(\frac{m_{h_{2}}}{m_{h_{1}}}\right)^{2}}.$ (1)

The partial decay width for $h_{1}\to aa$ is the same as Eq. (1) except for
changing the notation $h_{2}$ to $a$.

In our investigation, we concentrate on final states characterized by multiple
photons and explore the charge-parity properties of light scalars and
pseudoscalars, with a specific focus on pairs of $h_{2}$ or $a$ produced from
$h_{1}$. These secondary particles then undergo distinct decay processes:
$h_{2}$ decays into $\gamma\gamma$ and $\pi^{0}\pi^{0}$ Ellis:2012sd ;
Ellis:2012zp ; Chang:2015sdy , while $a$ transforms into $\gamma\gamma$,
$\pi^{0}\pi^{0}\pi^{0}$ Dobrescu:2000jt , where the $\pi^{0}$ subsequently
decays into a pair of photons due to the chiral anomaly effect. To highlight
these photon-rich decay modes from $h_{2}$ and $a$, we consider a mass range
of approximately $0.45$ GeV $\lesssim m_{h_{2},a}\lesssim 1.0$ GeV, ensuring
their dominance in the overall decay processes.

In our study, we illustrate the distinctive characteristics of the Higgs boson
exotic decay process, specifically $h_{1}\rightarrow h_{2}h_{2}/aa\rightarrow
4\gamma$. An essential aspect of our investigation involves the determination
of the opening angle between two collimated photons arising from the decay of
either $h_{2}$ or $a$. This opening angle, denoted as $\Delta
R_{\gamma\gamma}$, can be estimated as $\Delta
R_{\gamma\gamma}\sim\frac{2}{\gamma_{h_{2},a}}\sim
4\frac{m_{h_{2},a}}{m_{h_{1}}}$, and it falls within the range of $0.015$ to
$0.035$ for $0.45$ GeV $\lesssim m_{h_{2},a}\lesssim 1.0$ GeV. Here,
$\gamma_{h_{2},a}\sim\frac{m_{h_{1}}}{2m_{h_{2},a}}$ denotes the Lorentz
factor of $h_{2}$ or $a$. Significantly, $\Delta R_{\gamma\gamma}\lesssim
0.04$ is approximately the same size as a typical energy cluster resulting
from a single photon within the ATLAS electromagnetic calorimeter (ECAL)
ATLAS:2018dfo . Consequently, the small opening angle between two photons in
our analysis poses a challenge for the existing ATLAS triggers, as they cannot
readily distinguish between an energy deposit in the ECAL arising from the
photon-jet and that from a single photon. Therefore, at the ATLAS trigger
level, our photon-jet is initially identified as a single photon-like object
and subsequently recorded for further analysis. The primary objective of our
work is to effectively differentiate such photon-jets from single photons or
neutral pions within the ATLAS ECAL using advanced machine learning
techniques. However, when $\Delta R_{\gamma\gamma}\gtrsim 0.3$, the two
photons can be treated as two isolated photons within the ATLAS detector
ATLAS:2015rsn . In general, photons with opening angles within the range of
$0.04\lesssim\Delta R_{\gamma\gamma}\lesssim 0.3$ can be categorized as
photon-jet objects. Unlike previous studies on photon-jets that consider
relatively larger $\Delta R_{\gamma\gamma}$ values with the aid of jet
substructure analysis Ellis:2012sd ; Ellis:2012zp ; Chakraborty:2017mbz ;
Wang:2021uyb , our focus in this work centers on the study of photon-jets with
smaller $\Delta R_{\gamma\gamma}$.

It’s worth noting that several factors influence the branching ratios of
$h_{2}$ and $a$ decay modes. Firstly, for $h_{2}\rightarrow\gamma\gamma$, its
branching ratio can be significantly enhanced if additional new vector-like
charged fermions or other charged scalar/vector particles are introduced.
These new particles contribute to loop processes in
$h_{2}\rightarrow\gamma\gamma$, resulting in an elevated branching ratio. The
same enhancement applies to $a\rightarrow\gamma\gamma$. Alternatively, if the
light pseudoscalar behaves as a gaugphilic ALP, the $a\rightarrow\gamma\gamma$
mode naturally becomes the dominant decay channel. Secondly, for
$m_{h_{2}}>2m_{\pi}$, the $h_{2}\rightarrow\pi\pi$ mode becomes the dominant
decay channel, with a fraction of
$\Gamma(h_{2}\rightarrow\pi^{+}\pi^{-})/\Gamma(h_{2}\rightarrow\pi^{0}\pi^{0})\sim
2$ Gunion:1989we . However, the light pseudoscalar can exhibit mixing with SM
CP-odd mesons if they share the same quantum numbers. For $m_{a}\lesssim
3m_{\pi}\approx 405$ MeV, the three-body decay modes,
$a\rightarrow\pi^{0}\gamma\gamma$, $2\pi^{0}\gamma$, $\pi^{+}\pi^{-}\gamma$,
are considerably suppressed due to phase space limitations, rendering the
branching ratio of $a\rightarrow\gamma\gamma$ nearly equal to one
Dobrescu:2000jt . Conversely, for $m_{a}\gtrsim 0.5$ GeV, isospin-violating
decay modes, $a\rightarrow 3\pi$, become accessible, competing with
$a\rightarrow\gamma\gamma$. Precise estimation of the decay width is
challenging due to QCD uncertainties. However, it’s important to note that the
branching fractions for $\eta$ meson decays into $\gamma\gamma$, $3\pi^{0}$,
and $\pi^{+}\pi^{-}\pi^{0}$ are approximately $39.41\%$, $32.68\%$, and
$22.92\%$ Workman:2022ynf , respectively111We ignore other decay modes from
$\eta$ which are less than $5\%$.. Since the $\eta$ meson and $a$ share the
same quantum numbers, the $a\rightarrow\gamma\gamma$ decay mode remains
significant, even for $m_{a}$ on the order of $1$ GeV.

In an effort to ensure the model independence of our studies, we
systematically set the decay branching ratios of
$h_{2}/a\rightarrow\gamma\gamma$, $h_{2}\rightarrow\pi^{0}\pi^{0}$, and
$a\rightarrow 3\pi^{0}$ to unity individually for each analysis. This approach
allows our findings to be easily rescaled according to the specific branching
ratios associated with each decay channel within any concrete Higgs portal
model, thereby enhancing the applicability of our results. Moreover, our focus
in this work remains exclusively on the prompt decays of $h_{2}$ and $a$,
deferring the study of long-lived $h_{2}$ and $a$ to future investigations.

##  3 Photon-jet identification with deep learning

We simulated a lead/liquid-argon (LAr) sampling electromagnetic calorimeter
(ECAL) with granularity similar to that of the ATLAS ECAL and a pseudorapidity
($\eta$) coverage of $-0.2<\eta<0.2$ using GEANT4 GEANT4:2002zbu . It consists
of a thin pre-sampling layer and three sampling layers longitudinal in the
shower depth. The first sampling layer is segmented into high-granularity
strips in the $\eta$ direction, with a cell size of $0.0031\times 0.098$ in
$\Delta\eta\times\Delta\phi$. The pre-sampling layer, second sampling layer
and third sampling layer have granularity of $0.025\times 0.1$, $0.025\times
0.0245$ and $0.05\times 0.0245$ in $\Delta\eta\times\Delta\phi$, respectively.
Throughout these simulations, we maintain a fixed value of zero for $\eta$.

We recorded the energy deposits of the photon-jets—which arise from the decay
of $h_{2}$ or $a$, the single photon, or $\pi^{0}\rightarrow\gamma\gamma$
backgrounds—in each cell of the calorimeter. Our study focuses on four
benchmark masses of $h_{2}$ and $a$; specifically, 0.45 GeV, 0.6 GeV, 0.8 GeV,
and 1 GeV. For each process, we generated a sample of 100,000 events with the
energy of the source particle uniformly distributed between 40 and 250 GeV.
For each sample, $70\%$ of the events were used as the training set and the
rest for the test set.

The three subsections that follow describe the our implementations of the
Boosted Decision Tree (BDT), Convolutional Neural Network (CNN), and Particle
Flow Network (PFN). Our full code is available at
https://github.com/womogenes/photon-jet. For each model class (BDT, CNN, PFN),
we trained three distinct models for the $h_{2}\rightarrow\pi^{0}\pi^{0}$,
$a\rightarrow\gamma\gamma$, and $a\rightarrow 3\pi^{0}$ tasks.

###  3.1 BDT implementation

We employed Gradient BDTs Friedman:2001wbq  with multi-class classification as
our first method for separating signal and background processes. BDTs are
composed of an ensemble of decision trees that iteratively build on each other
to generate outputs more accurate than each tree can produce on its own.

At a high level, training a Gradient BDT involves performing a number of
“boosting iterations,” where each iteration adds a new decision tree to the
ensemble. These new trees are intended to remediate mistakes made by earlier
trees, and have their initial parameters tuned by gradient descent. More
details can be found in Ref. Friedman:2001wbq .

Name |  Definition | Relevant ECAL layer
---|---|---
$f_{1}$ |  Fraction of energy in the first layer | First layer
$f_{\mathrm{side}}$ |  Fraction of energy outside core of three central strips, but within seven strips
$w_{s3}$ |  Lateral shower width, $\sqrt{\frac{\Sigma E_{i}(i-i_{\text{max}})^{2}}{\Sigma E_{i}}}$, $i$ runs over three strips around maximum strip
$w_{s20}$ |  Lateral shower width, $\sqrt{\frac{\Sigma E_{i}(i-i_{\text{max}})^{2}}{\Sigma E_{i}}}$, $i$ runs over 20 strips around maximum strip
$\Delta E_{s}$ |  Difference between the energy associated with the second maximum, and the energy reconstructed in the strip with the minimal value found between the first and second maxima
$E_{\mathrm{ratio}}$ |  Ratio of the energy difference associated with the largest and second largest energy deposits over the sum of these energies
$R_{\eta}$ |  Ratio in $\eta$ of cell energies in $3\times 7$ versus $7\times 7$ cell, both centered around the maxima | Second layer
$R_{\phi}$ |  Ratio in $\phi$ of cell energies in $3\times 3$ and $3\times 7$ cells, both centered around the maxima
$w_{\eta 2}$ |  Lateral shower width, $\sqrt{\frac{\Sigma E_{i}\eta_{i}^{2}}{\Sigma E_{i}}-(\frac{\Sigma E_{i}\eta_{i}}{\Sigma E_{i}})^{2}}$, $i$ runs over the cells within $3\times 5$ window around the maxima
Table 1: The shower shape variables used in the BDT studies.

The features we used as inputs to our BDTs are the shower shape variables
$f_{1}$, $f_{\mathrm{side}}$, $w_{s3}$, $w_{s20}$, $\Delta E_{s}$,
$E_{\mathrm{ratio}}$, $R_{\eta}$, $R_{\phi}$, $w_{\eta 2}$, as used in Ref.
ATLAS:2018fzd . The definitions of these variables are provided in Table 1,
and their distributions are visualized in Fig. 1.

We used sklearn.ensemble.GradientBoostingClassifier from the scikit-learn
Python package Pedregosa:2011ork  to implement our GBDTs. The most important
hyperparameters are described below:

  1. 1.

The depth of each decision tree is limited to a maximum of 5. We observed that
any higher values tended to result in overfitting of training data.

  2. 2.

The number of boosting stages is set to $100$, because after that we observed
very minimal performance gains.

  3. 3.

The learning rate is set to $0.5$, which means that after every boosting
stage, the contribution of the added decisions tree gets cut in half. Lower
values tended to result in underfitting or slower training while higher values
tended to result in overfitting.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 1:  The BDT variables of photon-jet from $a\rightarrow\gamma\gamma$
($m_{a}=1$ GeV) and relevant backgrounds from the single photon and the
neutral pion. The $a$, $\gamma$ and $\pi^{0}$ have energy in the range of
$[40,250]$ GeV.

###  3.2 CNN implementation

(a)

(b)

(c)

Figure 2:  The deposited energy per cell of (a) photon-jet from
$a\rightarrow\gamma\gamma$ ($m_{a}=1$ GeV) (b) $\gamma$ (c) $\pi^{0}$ at the
first layer of the ECAL. The $a$, $\gamma$ and $\pi^{0}$ have energy in the
range of $[40,250]$ GeV.

We show the energy deposits for all cells per ECAL layer represented as a 2D
image of dimension $N_{\text{cells}}(\phi)\times N_{\text{cells}}(\eta)$ in
Fig. 2, where $N_{\text{cells}}(\phi)$ and $N_{\text{cells}}(\eta)$ are the
number of cells in $\phi$ and $\eta$ direction, respectively. The value for
each cell represents the energy deposited in it. These images are used as the
inputs to our CNNs.

$4\times 16$32 channels3$\displaystyle\times$3 filter + ReLUMaxPool2D16
channels3$\displaystyle\times$3 filter + ReLUMaxPool2D2 fully connected
layers, ReLU10 nodes each$4\times 128$$16\times 16$$16\times 8$CNN
model$\underbrace{\ \ \ \ \ \ \ \ \ \ \ \ }$40 units2 fully connected
layersReLU, 32 nodes
each$P\left(\text{signal}\right)$$P\left(\pi^{0}\right)$$P(\gamma)$$\underbrace{\
\ \ \ \ \ \ \ \ \ \ \ }$CNN modelCNN modelCNN model Figure 3: Diagram of the
CNN architecture.

An illustration of the CNN architecture is shown in Fig. 3. Four separate CNN
models exist: one for each of the four ECAL layers. Each CNN model is
constructed with two convolutional layers with filters of size $3\times 3$ and
stride 1, with the rectified linear unit (ReLU) Agarap:2018uiz  as its
activation function. Each convolutional layer is followed by a max-pooling
layer of size $2\times 2$. A flatten layer is used to convert the 2D output
array from the pooling layer to 1D array. Finally, the 1D arrays from the four
CNN models are concatenated before being fed to two fully connected layers
with 32 nodes and the ReLU activation. The final output layer is a fully
connected layer with the softmax activation function and number of nodes equal
to the number of distinct classes providing a multi-class output. We used the
Tensorflow Abadi:2016kic  and Keras packages to implement our CNNs. For
training, we utilized the categorical cross-entropy loss function and the Adam
optimizer Kingma:2014vow .

###  3.3 PFN implementation

In addition to the BDT and CNN classification methods, we employed PFNs to
identify the photon-jets of our signal processes. Developed by Komiske et al.
Komiske:2018cqr , PFNs are a class of deep learning model that take as input a
jet represented in point-cloud form (i.e. an unordered set of feature vectors)
and output a vector of probabilities for classification. Komiske et al.
demonstrated that PFNs are able to model any permutation-invariant function on
a point cloud. That is to say, the order in which points are stored in the
point cloud does not impact the output of a PFN, and any function with this
property can be represented as a PFN222In Ref. Komiske:2018cqr , Komiske et
al. formulate a close relative of the PFN called the Energy Flow Network
(EFN). We considered using EFNs in our studies, but found that they are less
performant than PFNs because the former is essentially a sub-case of the
latter and less generalizable..

PFNs cannot directly take our ECAL images as input, so we must first
“devoxelize” the images into point clouds. To accomplish this, the array of
each sampling layer is transformed into an unordered set of “points” (feature
vectors), one for each cell in the sampling layer. Each point is a four-tuple
$(\eta,\phi,E,\ell)$ where:

  1. 1.

$\eta$ and $\phi$ are the original coordinates of the cell (normalized to have
mean zero and standard deviation 1),

  2. 2.

$E$ is the energy of the cell in GeV, and

  3. 3.

$\ell$ is the index of the layer (ranging from 1 to 4 for the pre-sampling to
third sampling layers).

This creates an unordered set of $960$ feature vectors (particles) for each
jet. As a final step, we filter out all cells with zero energy, as those
should have no effect on the model’s output and we can take greater advantage
of the variable-length nature of PFN inputs. This also helps compress data for
faster training and lower overfitting.

Once a jet has been devoxelized into a point cloud $\mathcal{C}$, it is
classified using the equation

$\text{PFN}(\mathcal{C})=F\\!\left(\sum_{\vec{p}\in\mathcal{C}}\Phi(\vec{p})\right)\\!,$
(2)

where $F$ and $\Phi$ are vector functions approximated by deep neural networks
(DNNs). Essentially, every particle in $\mathcal{C}$ is a four-dimensional
vector which gets mapped to a 128-dimensional “latent space” using the
function $\Phi:\mathbb{R}^{4}\rightarrow\mathbb{R}^{128}$. The latent mappings
of each particle are then summed up, which we implement using a
tf.math.reduce_sum layer in Tensorflow, and this 128-dimensional vector sum is
then converted to final classification probabilities using the function
$F:\mathbb{R}^{128}\rightarrow\mathbb{R}^{3}$.

$\underbrace{\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ }$$\Phi$8 layers$\text{Point
cloud}\
\mathcal{C}$$\overrightarrow{p_{2}}$$\ell$$E$$\phi$$\eta$$\mathbf{\Sigma}$128
units$\eta$$\phi$$E$$\ell$$\overrightarrow{p_{1}}$$\underbrace{\ \ \ \ \ \ \ \
\ \ \ \ \ }$$F$8
layers$P\left(\text{signal}\right)$$P\left(\pi^{0}\right)$$P(\gamma)$$\eta$$\phi$$E$$\ell$$\overrightarrow{p_{|\mathcal{C}|}}$
Figure 4: Diagram of the PFN architecture.

For our models, both $\Phi$ and $F$ are represented as DNNs with 8 hidden
layers of sizes $(256,256,256,256,128,128,128,128)$, with $\Phi$ having an
additional input layer of size 4 (the number of features per particle) and $F$
having an additional output layer of size 3 (the number of classes). The ReLU
activation function Agarap:2018uiz  is used in all layers except the output
layer of $F$, which uses softmax. A diagram of our PFN’s architecture is shown
in Fig. 4.

The categorical cross-entropy loss function and adam optimizer Kingma:2014vow
are used to train the weights of $F$ and $\Phi$. We implemented our model and
training using the TensorFlow Keras library Abadi:2016kic .

###  3.4 Performance

(a)

(b)

(c)

Figure 5:  The normalized confusion matrix for distinguishing between the
photon-jet produced in the process of (a) $h_{2}\rightarrow\pi^{0}\pi^{0}$ (b)
$a\rightarrow\gamma\gamma$ (c) $a\rightarrow 3\pi^{0}$, and the single photon
and $\pi^{0}$ backgrounds for the test set using CNN. The mass of $h_{2}$ and
$a$ is assumed to be 1 GeV.

Fig. 5 shows the confusion matrix for distinguishing between photon-jets
produced in three different signal processes and the same $\gamma$, $\pi^{0}$
backgrounds with the test set using CNN. The overall identification efficiency
of the photon-jet is above 99.2% with $\gamma$ rejection rate above 99.9% and
$\pi^{0}$ rejection rate above 99.8%.

(a)

(b)

(c)

Figure 6:  Analogous confusion matrices to those in Fig. 5 for PFNs instead of
CNNs.

An analogous confusion matrix for our PFNs is shown in Fig. 6. They outperform
our CNNs in signal identification efficiency but are behind in background
rejection rates. The average identification efficiency exceeds 99.7% (99.5%
for the $h_{2}\rightarrow\pi^{0}\pi^{0}$ task, 99.8% for the
$a\rightarrow\gamma\gamma$ task, and 99.9% for the $a\rightarrow 3\pi^{0}$
task), while its overall background rejection rate exceeds 99.8% for the
$\pi^{0}$ background and 99.9% for the $\gamma$.

Figure 7: Receiver operating characteristic (ROC) curve for all three model types on the $h_{2}\rightarrow\pi^{0}\pi^{0}$ identification task. Model | $\text{Rej}_{90\%}$ | $\text{Rej}_{95\%}$ | $\text{Rej}_{99\%}$ | $\text{Rej}_{99.5\%}$
---|---|---|---|---
CNN ($h_{2}\rightarrow\pi^{0}\pi^{0}$) | 19968 | 14976 | 327 | 39
BDT ($h_{2}\rightarrow\pi^{0}\pi^{0}$) | 144 | 41 | 5 | 3
PFN ($h_{2}\rightarrow\pi^{0}\pi^{0}$) | 2406 | 1399 | 802 | 573
CNN ($a\rightarrow\gamma\gamma$) | 19944 | 3324 | 20 | 11
BDT ($a\rightarrow\gamma\gamma$) | 188 | 58 | 7 | 4
PFN ($a\rightarrow\gamma\gamma$) | N/A | N/A | 1200 | 484
CNN ($a\rightarrow 3\pi^{0}$) | N/A | 59885 | 1619 | 166
BDT ($a\rightarrow 3\pi^{0}$) | 253 | 90 | 10 | 5
PFN ($a\rightarrow 3\pi^{0}$) | N/A | N/A | 1253 | 676
Table 2: Comparison of model efficiency. For each of our nine models (CNN,
BDT, and PFN trained across three tasks), we evaluated the rejection rates
(defined as 1 over background efficiency) at 90%, 95%, 99%, and 99.5% working
points. The rejection rate appears as N/A if a model performs so well that no
threshold makes it dip below the working point.

(a)

(b)

(c)

Figure 8:  Comparison of identification efficiency of photon-jets between CNNs
(orange), PFNs (green), and BDTs (blue) in the process of (a)
$h_{2}\rightarrow\pi^{0}\pi^{0}$ (b) $a\rightarrow\gamma\gamma$ (c)
$a\rightarrow 3\pi^{0}$. The top row represent signal signatures and the
bottom two rows show background signatures. The $y$-coordinate of each point
represents the proportion of events with a given energy ($x$-axis) that are
classified as signal, with associated error bars.

Fig. 7, Table 2, and Fig. 8 compare the efficiency and mis-tag rates of our
CNNs, BDTs, and PFNs across different working points and energy bins.
(“Efficiency” refers to the proportion of events tagged as signal, and “mis-
tag rate” refers to the proportion of background events tagged as signal.)

Our results indicate that BDTs are not well-suited to the photon-jet tasks, as
its signal efficiency remains consistently below that of CNNs and PFNs across
all energy bins. This could be explained by the fact that BDTs are less
structurally complex than other deep learning models, and therefore are
incapable of modeling the finer patterns present in photon-jets.

We observe that the CNNs have significantly higher efficiency on the
$h_{2}\rightarrow\pi^{0}\pi^{0}$ tagging task than $a\rightarrow\gamma\gamma$
and $a\rightarrow 3\pi^{0}$ tasks. Also, PFNs have better signal efficiencies
than CNNs, especially for the $a\rightarrow\gamma\gamma$ task. This lies
outside the statistically significant range of the error bars. However, PFNs
have worse mis-tag rates than CNNs, and this is especially pronounced for the
$h_{2}\rightarrow\pi^{0}\pi^{0}$ task. We will use the analysis results from
our CNNs in Sec. 4, though PFNs achieve similar levels of performance and
could be another suitable choice.

###  3.5 Interpretation of PFNs

We now seek to establish the interpretability of our PFNs, in hopes of
elucidating why it performs so well.

BDT variable |  $h_{2}\rightarrow\pi^{0}\pi^{0}$ PFN |  $a\rightarrow\gamma\gamma$ PFN |  $a\rightarrow 3\pi^{0}$ PFN
---|---|---|---
depth_weighted_total_e: Summed energy across all 960 calorimeter cells, directly weighted by layer (0 for pre-sampling layer, 1 for the first, 2 for the second, 3 for the third) |  $\Sigma_{117}$ (0.998) |  $\Sigma_{93}$ (0.998) |  $\Sigma_{73}$ (0.999)
total_e: Summed energy across all 960 calorimeter cells, unweighted |  $\Sigma_{88}$ (0.999) |  $\Sigma_{93}$ (0.997) |  $\Sigma_{19}$ (0.996)
depth_weighted_total_e2: Summed energy squared across all cells, directly weighted by layer |  $\Sigma_{68}$ (0.998) |  $\Sigma_{93}$ (0.996) |  $\Sigma_{73}$ (0.997)
prelayer_e: Summed energy across pre-sampling layer |  $\Sigma_{56}$ (0.847) |  $\Sigma_{7}$ (0.841) |  $\Sigma_{126}$ (0.876)
firstlayer_x2: Summed energy across first layer, weighted by $x$-coordinate squared $(\phi^{2})$ |  $\Sigma_{41}$ (0.834) |  $\Sigma_{78}$ (0.812) |  $\Sigma_{96}$ (0.907)
firstlayer_y2: Summed energy across first layer, weighted by $y$-coordinate squared $(\eta^{2})$ |  $\Sigma_{12}$ (0.820) |  $\Sigma_{57}$ (0.722) |  $\Sigma_{121}$ (0.887)
Table 3:  Physical interpretation of high-level features in PFN models.
Pearson correlation coefficients are given between six BDT features and their
most highly correlated $\Sigma$ units in each PFN. Only the six most prominent
BDT features are shown. Figure 9: Visual representation of Table 3. Scatter
plots of the activations of six select units in the $\Sigma$ layer of our
$a\rightarrow 3\pi^{0}$ PFN against correlated physical variables. Each point
represents a single sample event.

Many of the units in $\Sigma$—the layer of the PFN corresponding to the sum of
every particle’s latent space—correlate strongly with select BDT features
(input variables) which have physical interpretations. Refer to Table 3 for
the PFN units, their BDT feature counterparts, and the correlation value.
Additionally, see Fig. 9 for an analogous visual representation featuring each
of the 3,000 sample events plotted against the two variables (PFN unit and
physical variable).

Figure 10: For each unit in the $\Sigma$ layer of our $a\rightarrow 3\pi^{0}$
PFN, we plotted its SHAP value (how strongly it influenced the model’s
prediction) against its highest correlation with a BDT feature. The SHAP value
axis is on a log scale.

Using SHAPley values, a generalized way to explain the outputs of machine
learning models using game theory, we determined that the PFN units with
stronger physical variable analogs tended to have a larger impact on the
model’s output (see Fig. 10). Many of the units with strongest influence on
the PFN’s output also correlate strongly with a BDT feature.

There are other units that have weaker, yet still moderately strong influence
on the PFN’s output but which don’t correlate very strongly with any BDT
variables. Take, for instance $\Sigma_{26}$, whose best BDT feature
correlation is only $41.5\%$, but whose SHAP value of 1.04 exceeds that of
other units, many with even higher BDT feature correlations. This suggests
that the model learned to model other jet features not necessarily encompassed
by our BDT study, and perhaps even those not directly computable from ECAL
images. The PFN shows strong potential for being able to “recover” information
about the original photon-jet particle cloud that was lost in ECAL
measurements.

##  4 Results and implication to the Higgs portal model

(a)

(b)

(c)

Figure 11:  Feynman diagrams for photo-jet processes in the Higgs portal
models. Event toplologies: (a) $gg\rightarrow h_{1}\rightarrow
h_{2}h_{2}/aa\rightarrow(\gamma\gamma)(\gamma\gamma)$; (b) $gg\rightarrow
h_{1}\rightarrow h_{2}h_{2}\rightarrow(\pi^{0}\pi^{0})(\pi^{0}\pi^{0})$; (c)
$gg\rightarrow h_{1}\rightarrow
aa\rightarrow(\pi^{0}\pi^{0}\pi^{0})(\pi^{0}\pi^{0}\pi^{0})$. A photon-jet is
denoted by each pair of parentheses.

The predominant production channel for the SM-like Higgs boson is through
gluon fusion, with a cross section exceeding that of other production channels
by more than an order of magnitude. In the subsequent analysis, our attention
is directed specifically toward the gluon fusion production of $h_{1}$.
Subsequently, we explore the decay process of $h_{1}$ into a pair of $h_{2}$
or $a$. Specifically, to explore the implications of the Higgs portal models
in distinguishing photon-jet signatures from those of single photons and
neutral pions in the SM, we investigate three distinct signal topologies:

$\displaystyle gg\rightarrow h_{1}\rightarrow
h_{2}h_{2}/aa\rightarrow(\gamma\gamma)(\gamma\gamma),$ (3) $\displaystyle
gg\rightarrow h_{1}\rightarrow
h_{2}h_{2}\rightarrow(\pi^{0}\pi^{0})(\pi^{0}\pi^{0}),$ (4) $\displaystyle
gg\rightarrow h_{1}\rightarrow
aa\rightarrow(\pi^{0}\pi^{0}\pi^{0})(\pi^{0}\pi^{0}\pi^{0}).$ (5)

The corresponding Feynman diagrams for these signal processes are depicted in
Fig. 11. Notably, the highly collimated nature of the photons resulting from
the decays of $h_{2}$ and $a$ prevents them from satisfying conventional
photon isolation criteria. Consequently, novel signal signatures emerge,
including $2\gamma$-jets, $4\gamma$-jets, and $6\gamma$-jets, as opposed to
conventional multi-photon structures. We assume that the gluon fusion
production cross section of $h_{1}$ is consistent with the SM prediction,
denoted as $\sigma(gg\rightarrow h_{1})=54.67$ pb at $\sqrt{s}=14$ TeV
LHCHiggsCrossSectionWorkingGroup:2016ypw . Furthermore, to preserve the model
independence of our analysis, we consistently set the decay branching ratios
of $h_{2}/a\rightarrow\gamma\gamma$, $h_{2}\rightarrow\pi^{0}\pi^{0}$, and
$a\rightarrow 3\pi^{0}$ to unity for each investigation. As a result, the only
two free model parameters remaining are either
$\left(m_{h_{2}},|\mu_{h_{1}h_{2}h_{2}}|\right)$ or
$\left(m_{a},|\mu_{h_{1}aa}|\right)$. In this section, we intend to delineate
the sensitivity of photon-jet signatures to these two model parameters
individually.

We employ FeynRules Christensen:2008py  to generate the UFO model files
Degrande:2011ua  for the Higgs portal models under investigation. Our analysis
encompasses three major SM background processes: $gg\rightarrow
h_{1}\rightarrow\gamma\gamma$, $pp\rightarrow\gamma\gamma$ and
$pp\rightarrow\gamma j$333For the SM background $pp\rightarrow jj$, the
efficiency after basic event selections is extremely low. Thus, generating a
very large number of Monte Carlo background events (more than $10^{10}$) is
required, which is beyond the scope of this work.. To conduct this study, we
utilize MadGraph5_aMC@NLO Alwall:2014hca  with the NN23LO1 PDF set
NNPDF:2017mvq  to simulate the leading-order contributions of the signal
processes described in Eqs. (3) to (5) and the three SM background processes
occurring in $pp$ collisions at $\sqrt{s}=14$ TeV. The cross section for the
gluon fusion production of $h_{1}$ has been rescaled with N3LO QCD and NLO EW
accuracies LHCHiggsCrossSectionWorkingGroup:2016ypw . We applied ME-PS
matching with the MLM prescription Mangano:2006rw ; Alwall:2007fs  to
$pp\rightarrow\gamma\gamma$ and $pp\rightarrow\gamma j$, including the
emission of up to two additional partons for these two processes.
Additionally, Monte Carlo events are generated from MadGraph5_aMC@NLO for both
the signal and background processes. All events undergo parton showering,
hadronization, and the treatment of unstable particle decays through Pythia8
Bierlich:2022pfr . Our simulation culminates with the identification of
photon-jet signatures originating from either $h_{2}$ or $a$, a task carried
out using an ATLAS-like ECAL. Notably, the electromagnetic showers are
simulated with GEANT4, as outlined in Sec. 3.

Following the Pythia8 simulations, we apply specific criteria to the generated
truth-level events for both signal and background processes in order to
identify photon-jet candidates. These criteria are intended to select events
featuring at least two photon-jet candidates, which are defined as follows. We
employ the anti-$k_{T}$ jet clustering algorithm Cacciari:2008gp  with
$R_{J}<0.25$ to group collimated photons, where $R_{J}$ denotes the jet cone
radius. Given that photon-jet candidates primarily deposit their energy in the
ECAL rather than the HCAL associated with ordinary QCD jets, we require
$\log\theta_{J}<-0.8$, where $\theta_{J}$ represents the hadronic energy
fraction of a jet. Within the truth-level events, the hadronic energy fraction
encompasses all charged and neutral hadrons, excluding $\pi^{0}$ and $\eta$
mesons in our analysis. Both $\pi^{0}$ and $\eta$ will dominantly decay to a
pair of photons, which deposit almost all of their energy in the ECAL instead
of the HCAL. Furthermore, within the jet cone of $R_{J}<0.25$, we veto all
charged tracks with $P_{T}>2$ GeV to mitigate the possibility of QCD jets
mimicking photon-jet candidates. On the other hand, to enforce a loose track
isolation criterion for the photon-jet candidates, they must be isolated from
nearby charged tracks within a cone radius of $\Delta R<0.2$, where the scalar
sum of the $P_{T}$ of tracks is required to be smaller than $5\%$ of the
photon-jet candidate’s $P_{T}$. Since two photon-jet candidates originate from
the SM-like Higgs boson through gluon fusion in this study, they are expected
to be energetic and distributed in the central region. Consequently, we
require that the leading photon-jet, $J_{1}$, and the sub-leading photon-jet,
$J_{2}$, meet the criteria of $P_{T}(J_{1,2})>40$ GeV and $|\eta_{J}|<2.5$.
Additionally, we utilize the invariant mass distribution of the two photon-jet
candidates ($M_{J_{1}J_{2}}$), which can be reconstructed to align with the
SM-like Higgs boson mass in the signal processes and $gg\rightarrow
h_{1}\rightarrow\gamma\gamma$. However, this distribution displays a smooth,
decreasing profile for $pp\rightarrow\gamma\gamma$ and $pp\rightarrow\gamma
j$. To bolster the suppression of SM background events while preserving most
signal events, we apply the following event selection criteria:
$P_{T}(J_{1})>0.4M_{J_{1}J_{2}}$, $P_{T}(J_{2})>0.3M_{J_{1}J_{2}}$, and $120$
GeV $<M_{J_{1}J_{2}}<130$ GeV.

Following the event selections of the photon-jet candidates as described above
at the truth level, we employ advanced machine learning techniques to further
differentiate these photon-jet signatures from those originating from SM
background events. As elucidated in Sec. 3, both CNN and PFN emerge as the
potent tools for the discrimination of photon-jet signatures and the rejection
of SM background events. We consequently utilize the identification
efficiencies for photon-jet signatures and SM backgrounds achieved through the
CNN as an example to determine the event counts at the reconstructed level. It
is important to note that these efficiencies are assumed to be independent of
the variable $\eta$. Given that our two photon-jet candidates are
predominantly distributed in the central region, this assumption remains valid
for our analysis.

(a)

(b)

Figure 12:  The projection of limits at $95\%$ CL for (a)
$(m_{h_{2}},|\mu_{h_{1}h_{2}h_{2}}|)$ and (b) $(m_{a},|\mu_{h_{1}aa}|)$ for
various photon-jet signatures in Higgs portal models at the LHC with
$\sqrt{s}=14$ TeV and an integrated luminosity of $3000\,\textrm{fb}^{-1}$
without the pileup effect. The top gray bulk is the constraint from the exotic
decay branching ratios of the SM-like Higgs boson,
$B(h_{1}\rightarrow\text{undetected})<19\%$ ATLAS:2020qdt  and the gray line
is the constraint from the CMS search ($13$ TeV, $136$ fb-1) of exotic Higgs
boson decays $h_{1}\rightarrow h_{2}h_{2}/aa\rightarrow 4\gamma$ CMS:2022wpu ,
respectively.

We can then translate these acceptances, derived from the event selections and
CNN identification efficiencies, into constraints on the Higgs portal model
parameters $\left(m_{h_{2}},|\mu_{h_{1}h_{2}h_{2}}|\right)$ and
$\left(m_{a},|\mu_{h_{1}aa}|\right)$, depending on whether the light scalar is
CP-even or CP-odd. The projections of the $95\%$ confidence level (CL) limits
for $|\mu_{h_{1}h_{2}h_{2}}|$ (left) and $|\mu_{h_{1}aa}|$ (right) within the
range $0.45$ GeV $\leqslant m_{h_{2},a}\leqslant 1.0$ GeV, without the pileup
effect, for $\sqrt{s}=14$ TeV and an integrated luminosity of ${\cal{L}}=3000$
fb-1, are presented in Fig. 12. For the sake of comparison, we incorporate the
constraint from the exotic decay branching ratios of the SM-like Higgs boson,
$B(h_{1}\rightarrow\text{undetected})<19\%$ ATLAS:2020qdt , which arises from
data at $\sqrt{s}=13$ TeV and an integrated luminosity of ${\cal{L}}=139$
fb-1, into Fig. 12. Here $B(h_{1}\rightarrow\text{undetected})$ is defined as
$\Gamma_{\text{BSM}}/\left(\Gamma_{\text{BSM}}+\Gamma_{\text{SM}}\right)$ with
$\Gamma_{\text{BSM}}=\Gamma(h_{1}\rightarrow h_{2}h_{2})$ or
$\Gamma(h_{1}\rightarrow aa)$ and $\Gamma_{\text{SM}}=4.03$ MeV
LHCHiggsCrossSectionWorkingGroup:2011wcg . This constraint can be translated
to $|\mu_{h_{1}h_{2}h_{2}}|\lesssim 3.34$ GeV and $|\mu_{h_{1}aa}|\lesssim
3.34$ GeV, respectively.

Finally, we conduct a comparison of our findings with previous photon-jet
searches conducted by both the ATLAS and CMS Collaborations. The ATLAS
Collaboration has searched for similar signatures in three distinct
publications ATLAS:2012soa ; ATLAS:2018dfo ; ATLAS:2023eet . In the first one
ATLAS:2012soa , the investigation centers around Higgs boson exotic decays to
pairs of light pseudoscalars with masses below $0.4$ GeV. This scenario
involves largely boosted pseudoscalars, making the final state challenging to
distinguish from a single-photon signature, thus serving as a valuable
complement to our study. The second one ATLAS:2018dfo  explores high-mass
resonance scalar particles with masses exceeding $200$ GeV, differing from our
focus on the SM-like Higgs boson mass. Consequently, a direct comparison with
our results is not feasible. In the third one ATLAS:2023eet , the
investigation involves Higgs boson exotic decays to pairs of axion-like
particles with masses exceeding $5$ GeV, providing another complementary
aspect to our study. The CMS Collaboration, in their study CMS:2022wpu ,
explores Higgs boson exotic decays to pairs of light scalars with masses
ranging from $0.1$ GeV to $1.2$ GeV, aligning with the scope of our research.
Consequently, we incorporate these constraints into Fig. 12 for comparison.
Notably, our advanced machine learning techniques enable the exploration of
much smaller values (${\cal{O}}(1)$ MeV) for $|\mu_{h_{1}h_{2}h_{2}}|$ and
$|\mu_{h_{1}aa}|$ at the High-Luminosity LHC.

##  5 Conclusion

The Higgs boson, the final component of the Standard Model (SM), was
discovered over ten years ago. Various properties of this SM-like Higgs boson
have been precisely measured, including its mass, width, charge-parity (CP),
and couplings to gauge bosons and the third generation of fermions. In future
studies, measurements of the Higgs boson self-coupling, its coupling to the
second generation of fermions, and other properties can shed light on the
origin of electroweak symmetry breaking and the mass origin of matter. Beyond
examining these fundamental properties of the scalar boson, a crucial aspect
involves directly investigating its potential connection to new physics. This
linkage between the SM sector and the dark sector is known as the Higgs
portal. In models where new particles have masses less than half of the SM-
like Higgs boson within the Higgs portal, the Large Hadron Collider (LHC)
provides an opportunity to explore exotic and invisible decays of the Higgs
boson. Therefore, the search for these potential new decay channels of the SM-
like Higgs boson at the LHC represents a powerful approach to unravel the
mysteries of new physics.

We specifically investigate a particular aspect of Higgs boson exotic decays,
focusing on the photon-jet signatures at the LHC in this study. The unique
photon-jet signautre arises when a few collimated photons are generated,
failing to satisfy the standard photon isolation criteria. If a pair of new
light particles is produced from the SM-like Higgs boson, primarily decaying
to photons in the final state, two highly boosted light particles result in
the natural generation of two photon-jets. Due to the photon-jet’s predominant
energy deposition in the electromagnetic calorimeter (ECAL), it exhibits
behavior akin to a single photon and neutral pion in the SM. To discern the
photon-jet signature from SM backgrounds, we employ advanced machine learning
techniques in this investigation. This work encompasses three photon-jet
signals: $h_{2}/a\rightarrow\gamma\gamma$, $h_{2}\rightarrow\pi^{0}\pi^{0}$,
and $a\rightarrow 3\pi^{0}$, where $h_{2}$ ($a$) represents a light scalar
(pseudoscalar). Our findings, depicted in Fig. 8, indicate that both
Convolutional Neural Networks (CNN) and Particle Flow Networks (PFN) serve as
potent tools for distinguishing photon-jet signatures from SM backgrounds,
such as the single photon and neutral pion from QCD jets. Taking CNN as an
example, the photon-jet can be identified with an efficiency exceeding $99\%$,
accompanied by a background rejection rate surpassing $99\%$. Notably, our
comparative analysis reveals that the two deep learning methodologies, CNN and
PFN, outperform the traditional machine learning analysis, Boosted Decision
Trees (BDT), as illustrated in Fig. 7 and Table 2.

With the aid of basic event selections and deep learning analysis, we can
forecast future constraints on the $h_{1}-h_{2}-h_{2}$ and $h_{1}-a-a$ three-
point interactions within Higgs portal models, as denoted by the couplings
$\mu_{h_{1}h_{2}h_{2}}$ and $\mu_{h_{1}aa}$, both possessing dimensions
equivalent to mass. Notably, investigating exotic decays of the Higgs boson
proves to be the most suitable approach for detecting the magnitudes of these
three-point interactions. Projections of the $95\%$ Confidence Level (CL)
limits for $|\mu_{h_{1}h_{2}h_{2}}|$ and $|\mu_{h_{1}aa}|$ at the High-
Luminosity LHC, without pileup effects, are presented in Fig. 12. Our findings
reveal that $2\sigma$ bounds for $|\mu_{h_{1}h_{2}h_{2}}|$ and
$|\mu_{h_{1}aa}|$ can be explored down to the order of ${\cal{O}}(1)$ MeV,
demonstrating the efficacy of CNN and PFN in achieving this level of
precision.

###### Acknowledgements.

The work of S.-C. Hsu and K. Li are supported by the U.S. Department of
Energy, Office of Science, Office of Early Career Research Program under Award
number DE-SC0015971. The work of C.-T. Lu is supported by the National Natural
Science Foundation of China (NNSFC) under grant No. 12335005.

## References

  * (1) A. Tumasyan et al. [CMS], Nature 607, no.7917, 60-68 (2022) doi:10.1038/s41586-022-04892-x [arXiv:2207.00043 [hep-ex]].
  * (2) G. Aad et al. [ATLAS], Nature 607, no.7917, 52-59 (2022) [erratum: Nature 612, no.7941, E24 (2022)] doi:10.1038/s41586-022-04893-w [arXiv:2207.00092 [hep-ex]].
  * (3) G. Arcadi, A. Djouadi and M. Raidal, Phys. Rept. 842, 1-180 (2020) doi:10.1016/j.physrep.2019.11.003 [arXiv:1903.03616 [hep-ph]].
  * (4) V. De Romeri, D. Karamitros, O. Lebedev and T. Toma, JHEP 10, 137 (2020) doi:10.1007/JHEP10(2020)137 [arXiv:2003.12606 [hep-ph]].
  * (5) S. Matsumoto, Y. L. S. Tsai and P. Y. Tseng, JHEP 07, 050 (2019) doi:10.1007/JHEP07(2019)050 [arXiv:1811.03292 [hep-ph]].
  * (6) K. Bondarenko, A. Boyarsky, T. Bringmann, M. Hufnagel, K. Schmidt-Hoberg and A. Sokolenko, JHEP 03, 118 (2020) doi:10.1007/JHEP03(2020)118 [arXiv:1909.08632 [hep-ph]].
  * (7) T. Cohen, D. E. Morrissey and A. Pierce, Phys. Rev. D 86, 013009 (2012) doi:10.1103/PhysRevD.86.013009 [arXiv:1203.2924 [hep-ph]].
  * (8) W. Huang, J. Shu and Y. Zhang, JHEP 03, 164 (2013) doi:10.1007/JHEP03(2013)164 [arXiv:1210.0906 [hep-ph]].
  * (9) W. Chao, Phys. Rev. D 92, no.1, 015025 (2015) doi:10.1103/PhysRevD.92.015025 [arXiv:1412.3823 [hep-ph]].
  * (10) F. Bezrukov and D. Gorbunov, JHEP 05, 010 (2010) doi:10.1007/JHEP05(2010)010 [arXiv:0912.0390 [hep-ph]].
  * (11) F. Bezrukov and A. Keats, Phys. Rev. D 104, no.7, 075020 (2021) doi:10.1103/PhysRevD.104.075020 [arXiv:2108.02540 [hep-ph]].
  * (12) B. Batell, M. Low, E. T. Neil and C. B. Verhaaren, [arXiv:2203.05531 [hep-ph]].
  * (13) R. E. Shrock and M. Suzuki, Phys. Lett. B 110, 250 (1982) doi:10.1016/0370-2693(82)91247-3
  * (14) D. Curtin, R. Essig, S. Gori, P. Jaiswal, A. Katz, T. Liu, Z. Liu, D. McKeen, J. Shelton and M. Strassler, et al. Phys. Rev. D 90, no.7, 075004 (2014) doi:10.1103/PhysRevD.90.075004 [arXiv:1312.4992 [hep-ph]].
  * (15) A. Chakraborty, A. M. Iyer and T. S. Roy, Nucl. Phys. B 932, 439-470 (2018) doi:10.1016/j.nuclphysb.2018.05.019 [arXiv:1707.07084 [hep-ph]].
  * (16) J. Alimena, J. Beacham, M. Borsato, Y. Cheng, X. Cid Vidal, G. Cottin, A. De Roeck, N. Desai, D. Curtin and J. A. Evans, et al. J. Phys. G 47, no.9, 090501 (2020) doi:10.1088/1361-6471/ab4574 [arXiv:1903.04497 [hep-ex]].
  * (17) G. Albouy, J. Barron, H. Beauchesne, E. Bernreuther, M. Bona, C. Cazzaniga, C. Cesarotti, T. Cohen, A. de Cosa and D. Curtin, et al. Eur. Phys. J. C 82, no.12, 1132 (2022) doi:10.1140/epjc/s10052-022-11048-8 [arXiv:2203.09503 [hep-ph]].
  * (18) R. Franceschini, D. Kim, K. Kong, K. T. Matchev, M. Park and P. Shyamsundar, Rev. Mod. Phys. 95, no.4, 045004 (2023) doi:10.1103/RevModPhys.95.045004 [arXiv:2206.13431 [hep-ph]].
  * (19) B. A. Dobrescu, G. L. Landsberg and K. T. Matchev, Phys. Rev. D 63, 075003 (2001) doi:10.1103/PhysRevD.63.075003 [arXiv:hep-ph/0005308 [hep-ph]].
  * (20) N. Toro and I. Yavin, Phys. Rev. D 86, 055005 (2012) doi:10.1103/PhysRevD.86.055005 [arXiv:1202.6377 [hep-ph]].
  * (21) N. Steinberg and J. D. Wells, JHEP 08, 120 (2021) doi:10.1007/JHEP08(2021)120 [arXiv:2101.00520 [hep-ph]].
  * (22) A. Alves, A. G. Dias and D. D. Lopes, JHEP 10, 012 (2021) doi:10.1007/JHEP10(2021)012 [arXiv:2105.01095 [hep-ph]].
  * (23) S. Knapen, S. Kumar and D. Redigolo, Phys. Rev. D 105, no.11, 115012 (2022) doi:10.1103/PhysRevD.105.115012 [arXiv:2112.07720 [hep-ph]].
  * (24) S. D. Ellis, T. S. Roy and J. Scholtz, Phys. Rev. Lett. 110, no.12, 122003 (2013) doi:10.1103/PhysRevLett.110.122003 [arXiv:1210.1855 [hep-ph]].
  * (25) S. D. Ellis, T. S. Roy and J. Scholtz, Phys. Rev. D 87, no.1, 014015 (2013) doi:10.1103/PhysRevD.87.014015 [arXiv:1210.3657 [hep-ph]].
  * (26) F. Domingo, JHEP 03, 052 (2017) doi:10.1007/JHEP03(2017)052 [arXiv:1612.06538 [hep-ph]].
  * (27) D. Wang, L. Wu, J. M. Yang and M. Zhang, Phys. Rev. D 104, no.9, 095016 (2021) doi:10.1103/PhysRevD.104.095016 [arXiv:2102.01532 [hep-ph]].
  * (28) [ATLAS], ATLAS-CONF-2012-079.
  * (29) M. Aaboud et al. [ATLAS], Phys. Rev. D 99, no.1, 012008 (2019) doi:10.1103/PhysRevD.99.012008 [arXiv:1808.10515 [hep-ex]].
  * (30) [CMS], CMS-PAS-HIG-21-016.
  * (31) [ATLAS], ATLAS-CONF-2023-040.
  * (32) J. Ren, D. Wang, L. Wu, J. M. Yang and M. Zhang, JHEP 11, 138 (2021) doi:10.1007/JHEP11(2021)138 [arXiv:2106.07018 [hep-ph]].
  * (33) D. Wang, J. H. Cho, J. Kim, S. Lee, P. Sanyal and J. Song, Phys. Rev. D 109, no.1, 015017 (2024) doi:10.1103/PhysRevD.109.015017 [arXiv:2310.17741 [hep-ph]].
  * (34) J. Chang, K. Cheung and C. T. Lu, Phys. Rev. D 93, no.7, 075013 (2016) doi:10.1103/PhysRevD.93.075013 [arXiv:1512.06671 [hep-ph]].
  * (35) B. Sheff, N. Steinberg and J. D. Wells, Phys. Rev. D 104, no.3, 036009 (2021) doi:10.1103/PhysRevD.104.036009 [arXiv:2008.10568 [hep-ph]].
  * (36) P. Draper and D. McKeen, Phys. Rev. D 85, 115023 (2012) doi:10.1103/PhysRevD.85.115023 [arXiv:1204.1061 [hep-ph]].
  * (37) L. Aparicio, A. Azatov, E. Hardy and A. Romanino, JHEP 05, 077 (2016) doi:10.1007/JHEP05(2016)077 [arXiv:1602.00949 [hep-ph]].
  * (38) U. Ellwanger and C. Hugonie, JHEP 05, 114 (2016) doi:10.1007/JHEP05(2016)114 [arXiv:1602.03344 [hep-ph]].
  * (39) F. Domingo, S. Heinemeyer, J. S. Kim and K. Rolbiecki, Eur. Phys. J. C 76, no.5, 249 (2016) doi:10.1140/epjc/s10052-016-4080-2 [arXiv:1602.07691 [hep-ph]].
  * (40) C. W. Chiang, H. Fukuda, M. Ibe and T. T. Yanagida, Phys. Rev. D 93, no.9, 095016 (2016) doi:10.1103/PhysRevD.93.095016 [arXiv:1602.07909 [hep-ph]].
  * (41) C. T. Lu, Phys. Rev. D 108, no.11, 115029 (2023) doi:10.1103/PhysRevD.108.115029 [arXiv:2210.15648 [hep-ph]].
  * (42) S. D. Lane, H. S. Lee and I. M. Lewis, [arXiv:2305.00013 [hep-ph]].
  * (43) B. P. Roe, H. J. Yang, J. Zhu, Y. Liu, I. Stancu and G. McGregor, Nucl. Instrum. Meth. A 543, no.2-3, 577-584 (2005) doi:10.1016/j.nima.2004.12.018 [arXiv:physics/0408124 [physics]].
  * (44) V. Ayyar, W. Bhimji, L. Gerhardt, S. Robertson and Z. Ronaghi, EPJ Web Conf. 245, 06003 (2020) doi:10.1051/epjconf/202024506003 [arXiv:2002.05761 [hep-ex]].
  * (45) P. T. Komiske, E. M. Metodiev and J. Thaler, JHEP 01, 121 (2019) doi:10.1007/JHEP01(2019)121 [arXiv:1810.05165 [hep-ph]].
  * (46) A. Filimonova, R. Schäfer and S. Westhoff, Phys. Rev. D 101, no.9, 095006 (2020) doi:10.1103/PhysRevD.101.095006 [arXiv:1911.03490 [hep-ph]].
  * (47) F. Abudinén et al. [Belle-II], Phys. Rev. Lett. 125, no.16, 161806 (2020) doi:10.1103/PhysRevLett.125.161806 [arXiv:2007.13071 [hep-ex]].
  * (48) T. Ferber, A. Filimonova, R. Schäfer and S. Westhoff, JHEP 04, 131 (2023) doi:10.1007/JHEP04(2023)131 [arXiv:2201.06580 [hep-ph]].
  * (49) I. Adachi et al. [Belle-II], Phys. Rev. D 108, no.11, L111104 (2023) doi:10.1103/PhysRevD.108.L111104 [arXiv:2306.02830 [hep-ex]].
  * (50) J. Blumlein, J. Brunner, H. J. Grabosch, P. Lanius, S. Nowak, C. Rethfeldt, H. E. Ryseck, M. Walter, D. Kiss and Z. Jaki, et al. Z. Phys. C 51, 341-350 (1991) doi:10.1007/BF01548556
  * (51) B. Döbrich, J. Jaeckel, F. Kahlhoefer, A. Ringwald and K. Schmidt-Hoberg, JHEP 02, 018 (2016) doi:10.1007/JHEP02(2016)018 [arXiv:1512.03069 [hep-ph]].
  * (52) J. L. Feng, I. Galon, F. Kling and S. Trojanowski, Phys. Rev. D 98, no.5, 055021 (2018) doi:10.1103/PhysRevD.98.055021 [arXiv:1806.02348 [hep-ph]].
  * (53) L. Harland-Lang, J. Jaeckel and M. Spannowsky, Phys. Lett. B 793, 281-289 (2019) doi:10.1016/j.physletb.2019.04.045 [arXiv:1902.04878 [hep-ph]].
  * (54) D. Gorbunov, I. Krasnov and S. Suvorov, Phys. Lett. B 820, 136524 (2021) doi:10.1016/j.physletb.2021.136524 [arXiv:2105.11102 [hep-ph]].
  * (55) F. Kling, S. Li, H. Song, S. Su and W. Su, JHEP 08, 001 (2023) doi:10.1007/JHEP08(2023)001 [arXiv:2212.06186 [hep-ph]].
  * (56) J. Liu, Y. Luo and M. Song, JHEP 09, 104 (2023) doi:10.1007/JHEP09(2023)104 [arXiv:2304.05435 [hep-ph]].
  * (57) S. Baek, P. Ko and W. I. Park, Phys. Lett. B 747, 255-259 (2015) doi:10.1016/j.physletb.2015.06.002 [arXiv:1407.6588 [hep-ph]].
  * (58) M. J. Dolan, F. Kahlhoefer, C. McCabe and K. Schmidt-Hoberg, JHEP 03, 171 (2015) [erratum: JHEP 07, 103 (2015)] doi:10.1007/JHEP03(2015)171 [arXiv:1412.5174 [hep-ph]].
  * (59) M. J. Dolan, T. Ferber, C. Hearty, F. Kahlhoefer and K. Schmidt-Hoberg, JHEP 12, 094 (2017) [erratum: JHEP 03, 190 (2021)] doi:10.1007/JHEP12(2017)094 [arXiv:1709.00009 [hep-ph]].
  * (60) A. Bharucha, F. Brümmer, N. Desai and S. Mutzel, JHEP 02, 141 (2023) doi:10.1007/JHEP02(2023)141 [arXiv:2209.03932 [hep-ph]].
  * (61) D. K. Ghosh, A. Ghoshal and S. Jeesun, JHEP 01, 026 (2024) doi:10.1007/JHEP01(2024)026 [arXiv:2305.09188 [hep-ph]].
  * (62) J. Liu, X. Ma, L. T. Wang and X. P. Wang, Phys. Rev. D 107, no.9, 095016 (2023) doi:10.1103/PhysRevD.107.095016 [arXiv:2210.09335 [hep-ph]].
  * (63) C. H. V. Chang, C. R. Chen, S. Y. Ho and S. Y. Tseng, Phys. Rev. D 104, no.1, 015030 (2021) doi:10.1103/PhysRevD.104.015030 [arXiv:2102.05012 [hep-ph]].
  * (64) M. Bauer, M. Heiles, M. Neubert and A. Thamm, Eur. Phys. J. C 79, no.1, 74 (2019) doi:10.1140/epjc/s10052-019-6587-9 [arXiv:1808.10323 [hep-ph]].
  * (65) M. W. Winkler, Phys. Rev. D 99, no.1, 015018 (2019) doi:10.1103/PhysRevD.99.015018 [arXiv:1809.01876 [hep-ph]].
  * (66) M. Aaboud et al. [ATLAS], Phys. Rev. D 99, no.1, 012008 (2019) doi:10.1103/PhysRevD.99.012008 [arXiv:1808.10515 [hep-ex]].
  * (67) G. Aad et al. [ATLAS], Eur. Phys. J. C 76, no.4, 210 (2016) doi:10.1140/epjc/s10052-016-4034-8 [arXiv:1509.05051 [hep-ex]].
  * (68) J. F. Gunion, H. E. Haber, G. L. Kane and S. Dawson, Front. Phys. 80, 1-404 (2000) SCIPP-89/13.
  * (69) R. L. Workman et al. [Particle Data Group], PTEP 2022, 083C01 (2022) doi:10.1093/ptep/ptac097
  * (70) S. Agostinelli et al. [GEANT4], Nucl. Instrum. Meth. A 506, 250-303 (2003) doi:10.1016/S0168-9002(03)01368-8
  * (71) J. H. Friedman, Annals Statist. 29, no.5, 1189-1232 (2001) doi:10.1214/aos/1013203451
  * (72) M. Aaboud et al. [ATLAS], Eur. Phys. J. C 79, no.3, 205 (2019) doi:10.1140/epjc/s10052-019-6650-6 [arXiv:1810.05087 [hep-ex]].
  * (73) F. Pedregosa, G. Varoquaux, A. Gramfort, O. Grisel, M. Blondel, A. Müller, J. Nothman, G. Louppe, P. Prettenhofer and R. Weiss, et al. J. Machine Learning Res. 12, 2825-2830 (2011) [arXiv:1201.0490 [cs.LG]].
  * (74) A. F. Agarap, [arXiv:1803.08375 [cs.NE]].
  * (75) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean and M. Devin, et al. [arXiv:1603.04467 [cs.DC]].
  * (76) D. P. Kingma and J. Ba, [arXiv:1412.6980 [cs.LG]].
  * (77) P. T. Komiske, E. M. Metodiev and J. Thaler, JHEP 01, 121 (2019) doi:10.1007/JHEP01(2019)121 [arXiv:1810.05165 [hep-ph]].
  * (78) D. de Florian et al. [LHC Higgs Cross Section Working Group], doi:10.23731/CYRM-2017-002 [arXiv:1610.07922 [hep-ph]].
  * (79) N. D. Christensen and C. Duhr, Comput. Phys. Commun. 180, 1614-1641 (2009) doi:10.1016/j.cpc.2009.02.018 [arXiv:0806.4194 [hep-ph]].
  * (80) C. Degrande, C. Duhr, B. Fuks, D. Grellscheid, O. Mattelaer and T. Reiter, Comput. Phys. Commun. 183, 1201-1214 (2012) doi:10.1016/j.cpc.2012.01.022 [arXiv:1108.2040 [hep-ph]].
  * (81) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli and M. Zaro, JHEP 07, 079 (2014) doi:10.1007/JHEP07(2014)079 [arXiv:1405.0301 [hep-ph]].
  * (82) R. D. Ball et al. [NNPDF], Eur. Phys. J. C 77, no.10, 663 (2017) doi:10.1140/epjc/s10052-017-5199-5 [arXiv:1706.00428 [hep-ph]].
  * (83) M. L. Mangano, M. Moretti, F. Piccinini and M. Treccani, JHEP 01, 013 (2007) doi:10.1088/1126-6708/2007/01/013 [arXiv:hep-ph/0611129 [hep-ph]].
  * (84) J. Alwall, S. Hoche, F. Krauss, N. Lavesson, L. Lonnblad, F. Maltoni, M. L. Mangano, M. Moretti, C. G. Papadopoulos and F. Piccinini, et al. Eur. Phys. J. C 53, 473-500 (2008) doi:10.1140/epjc/s10052-007-0490-5 [arXiv:0706.2569 [hep-ph]].
  * (85) C. Bierlich, S. Chakraborty, N. Desai, L. Gellersen, I. Helenius, P. Ilten, L. Lönnblad, S. Mrenna, S. Prestel and C. T. Preuss, et al. SciPost Phys. Codeb. 2022, 8 (2022) doi:10.21468/SciPostPhysCodeb.8 [arXiv:2203.11601 [hep-ph]].
  * (86) M. Cacciari, G. P. Salam and G. Soyez, JHEP 04, 063 (2008) doi:10.1088/1126-6708/2008/04/063 [arXiv:0802.1189 [hep-ph]].
  * (87) [ATLAS], ATLAS-CONF-2020-027.
  * (88) S. Dittmaier et al. [LHC Higgs Cross Section Working Group], doi:10.5170/CERN-2011-002 [arXiv:1101.0593 [hep-ph]].


# LLMs in Political Science: Heralding a New Era of Visual Analysis

Yu Wang<EMAIL_ADDRESS>Institute for Advanced Study in Social
Sciences, Fudan University, Shanghai, 200433, China

###### Abstract

Interest is increasing among political scientists in leveraging the extensive
information available in images. However, the challenge of interpreting these
images lies in the need for specialized knowledge in computer vision and
access to specialized hardware. As a result, image analysis has been limited
to a relatively small group within the political science community. This
landscape could potentially change thanks to the rise of large language models
(LLMs). This paper aims to raise awareness of the feasibility of using Gemini
for image content analysis. A retrospective analysis was conducted on a corpus
of 688 images. Content reports were elicited from Gemini for each image and
then manually evaluated by the authors. We find that Gemini is highly accurate
in performing object detection, which is arguably the most common and
fundamental task in image analysis for political scientists. Equally
important, we show that it is easy to implement as the entire command consists
of a single prompt in natural language; it is fast to run and should meet the
time budget of most researchers; and it is free to use and does not require
any specialized hardware. In addition, we illustrate how political scientists
can leverage Gemini for other image understanding tasks, including face
identification, sentiment analysis, and caption generation. Our findings
suggest that Gemini and other similar LLMs have the potential to drastically
stimulate and accelerate image research in political science and social
sciences more broadly.

###### keywords:

Gemini, Object Detection, Large Language Models, Image as Data

Political scientists are now in an era of data abundance, and machine learning
tools are increasingly used to extract meaning from data sets both textual and
visual [1]. While the primary emphasis has thus far centered on textual data
[2, 3], more and more political scientists are endeavoring to utilize image
data in their analyses [4]. Images play an important role in the political
world, whether it is election campaigns, military conflicts, immigration, or
poverty. As artificial intelligence continues to advance and its adoption
expands, political scientists are progressively utilizing AI to analyze the
composition and impact of visual materials in political contexts [5, 6].
Typically, this requires political scientists to have substantial training in
computer vision and machine learning more broadly. For example, researchers
need to understand concepts in computer vision such as the FAST Hessian
detector [7], SIFT features [8], R-CNN features [9]. Not to mention the fact
that many of these features and models nowadays require the use of specialized
hardware such as graphics processing units (GPUs). Stringent requirements of
computer vision expertise coupled with hard requirement on hardware have made
progress on image analysis in political science particularly slow. Even though
it is widely accepted that images contain vast amount of valuable information,
so far only a selected few political scientists are capable of carrying out
research in this field. Their research outputs, in turn, can only reach a
rather small audience due to the technicality of computer vision studies and
the difficulty in reproducing and further advancing the results.

This paper explores the potential of large language models (LLMs) for image
understanding tasks, with a focus on Gemini, which was released in December
2023 by Google. It demonstrates that zero-shot image annotation by Gemini has
an average rating of 3.8 on a 4-point Likert scale, with 1 being poor and 4
being excellent. While a few recent works have shown that LLMs can act as a
valuable tool in text annotation [10, 11], this paper shows how LLMs can
perform common image understanding tasks, such as object detection, face
identification and captioning, to facilitate image studies in political
science and social sciences more broadly. We focus on the task of object
detection because political scientists have a strong need to understand what
objects exist in the images for improved interpretability and downstream
modeling and because object detection has long been a bottleneck for political
scientists [5, 6]. We use 688 images from 33 news outlets. For each image, we
use Gemini to extract the objects contained in the image and grade the
identified objects from 1 to 4. Our results show that Gemini does a good job
in identifying objects from images. The quality is sufficient for downstream
tasks such as topic modeling [12] and ideology classification. Moreover,
throughout the entire process, the only command we used was a simple prompt
asking Gemini to retrieve objects, and we did not use any specialized computer
vision knowledge. This should help remove the technical barrier for all
political scientists. Moreover, the cost of using Gemini is zero, and it does
not require any specialized hardware on the researchers’ side. Lastly, in
terms of latency, it is relatively fast: it took 3,800 seconds to process 688
images or 5.5 seconds per image. This should meet the time budget of most
researchers. Researchers who require higher throughput could, for example, use
multiple processes.

## Results

We utilize the dataset from [5], which focuses on the Central American migrant
caravan and is recently published at the leading political science journal
Political Analysis. The dataset consists of 688 images collected from 33 news
outlets. The distribution of images among these outlets is uneven. For
example, there is one 1 from Axios and Buzzfeed each, 45 from USA Today and 46
from Vice News (Table 1).

Table 1: Average annotation ratings, rounded to the closest decimal, for all outlets across the entire political spectrum. The average ratings fall between 3 and 4 for all 33 news outlets. News Outlet | # Images | Political Leaning | Avg. Rating
---|---|---|---
ABC News | 26 | Left-Center | 3.9
Al Jazeera English | 38 | Center | 3.7
Associated Press | 25 | Center | 3.5
Axios | 1 | Center | 3.0
Bloomberg | 3 | Center | 3.7
Breitbart News | 31 | Right | 3.8
Business Insider | 45 | Center | 3.7
Buzzfeed | 1 | Left-Center | 4.0
CBS News | 20 | Left-Center | 4.0
CNBC | 43 | Center | 3.9
CNN | 25 | Left-Center | 3.7
Fortune | 6 | Not-Rated | 4.0
Fox News | 23 | Right-Center | 3.8
MSNBC | 2 | Left | 3.5
Mashable | 18 | Left | 3.9
NBC News | 11 | Left-Center | 3.9
National Review | 41 | Right | 3.9
New York Magazine | 20 | Left | 3.6
Newsweek | 34 | Left-Center | 3.4
Politico | 16 | Left-Center | 3.9
Reuters | 6 | Center | 3.8
The American Conservative | 5 | Right-Center | 3.0
The Hill | 22 | Center | 3.7
The Huffington Post | 31 | Left | 3.7
The New York Times | 18 | Left-Center | 3.7
The Verge | 4 | Left-Center | 4.0
The Wall Street Journal | 34 | Center | 3.7
The Washington Post | 12 | Left-Center | 4.0
The Washington Times | 18 | Right-Center | 3.9
Time | 15 | Left-Center | 3.9
USA Today | 45 | Center | 3.7
Vice News | 46 | Left | 3.6
Wired | 3 | Not-Rated | 4.0
All (weighted) | 688 |  | 3.8

We use Gemini to annotate each image and manually evaluate Gemini annotations
using a four-point Likert scale extending from 1 (poor), 2 (average), to 3
(good) and 4 (excellent) regarding their overall quality, with the
understanding that a grading of 3 would indicate that Gemini annotations are
potentially useful for conducting research on political images [13]. When
grading somewhat fits between two scores, we assign it the lower score to be
more conservative. Across the entire corpus, we observe that out of 688
images, 593 (86%) receive a rating of ‘excellent’, 49 (7%) receive a rating of
‘good’, 14 (2%) a rating of average and 32 (5%) ‘poor’. The average rating
across the entire corpus is 3.8.

In total, the model used 504 words (or phrases), the most frequent word is
‘person’, showing up in 349 out of 688 images, and the least frequent words,
such as ‘shipping container’, ‘surgical mask’ and ‘graduation cap’, are only
used once. Words that are often associated with the migrant caravan, such as
‘blanket’ (24 times) and ‘stroller’ (13 times), are also included in the
vocabulary. Interested readers could refer to the replication materials for
the details of the vocabulary and its distribution.

Table 2: For each given image, we first retrieve the objects using Gemini calls, and then evaluate the detected objects on a four-point Likert scale: poor, average, good, and excellent. Rating |  Objects |  Images
---|---|---
4: Excellent |  ‘person’: 1 ‘microphone’: 1 |
3: Good |  ‘person’: 1 ‘flag’: 1 ‘hat’: 1 ‘shirt’: 1 |
2: Average |  ‘city’: 3 ‘state’: 3 ‘country’: 3 ‘line’: 1 ‘map’: 1 |
1: Poor |  None |

We note that Gemini tends to perform particularly well with images with one or
two persons. Most of these images are leading U.S. politicians such as Donald
Trump, Nancy Pelosi, and Mike Pence. Gemini tends to perform poorly on images
that are either screenshots or figures with a large amount of text. In Table
2, we present some images, their respective ratings as well as detected
objects and their counts. To view the detected objects for all the images,
interested readers could refer to the replication materials.

## Discussion

Our study shows that Gemini excels in delivering highly accurate object
detection for images on the issue of Central American migrants. The results
from Gemini were accurate, capturing most objects in an image as well as
providing a reliable estimate of the count for each object. We conclude that
Gemini’s performance is impressive, particularly considering that its
annotations are zero-shot.

Note that we have mainly focused on object detection, because it is arguably
the most common and fundamental task for image analysis. Once detected, these
objects can then be used for downstream tasks such as clustering, topic
modeling, and ideology classification. This does not mean, however, that
Gemini’s capabilities are limited to object detection. In Figure 3, we further
illustrate how political scientists can leverage Gemini to retrieve
information about known politicians, sentiment undertone, and captioning of a
given image.

Table 3: Gemini can be used for various image tasks, including face identification, sentiment undertone detection, and captioning, all of which are of significant value to political scientists. Image Tasks |  Prompts |  Gemini | Images
---|---|---|---
Face identification |  List the known politicians in the image. |  The image shows Nancy Pelosi, the Speaker of the United States House of Representatives.
|
Sentiment undertone |  Use up to four words to describe the sentiment undertone of the image.

|  Serious, vigilant, watchful, cautious. |
Image captioning |  Generate a brief caption for the given image. |  A group of protesters hold signs and march down a street demanding the reunification of families that were separated at the border.
|

Besides its impressive performance, we note three other advantages of
leveraging Gemini for image research. One is that Gemini’s latency is fairly
low at 5.5 seconds per image, which translates to 650 images per hour. Another
is that Gemini is free to use. Researchers would not need access to
specialized hardware, such as high memory GPUs. A third advantage is that
calling Gemini requires minimal coding and machine learning expertise. All it
takes is a prompt in natural language that reflects the needs of the
researcher. We are confident that with the help of Gemini and other LLMs the
majority of political scientists should be able to conduct image analysis
whether their substantive interests lie in American politics, comparative
politics, or international relations.

## Materials and methods

### Dataset Characteristics

The analysis relies on a corpus of 688 images in 424 news articles from 33
news outlets covering the caravan of Central American migrants. In terms of
political leaning, the news outlets cover the entire spectrum, ranging from
Left (e.g., MSNBC) to Left-Center (e.g., CNN), Center (e.g., USA Today),
Right-Center (e.g., The Washington Times), and to Right (e.g., Breitbart
News). The news articles were published between October 3 and November 1, 2018
[5].

### Gemini Encoding and Prompting

Gemini was accessed online on January 14th, 2024, and operated as the gemini-
pro-vision version. Prompts were provided in a standardized format. Note that
we instruct Gemini to return the analysis results (1) as a one-line
dictionary, (2) in lowercase, and (3) in singular forms to facilitate easier
data analysis downstream.

  * •

List all the objects and people in the image, using lowercase and singular
forms. Include the count for each object. Please use the following one-line
format with curly brackets in the response: {“Object 1”: Count 1, “Object 2”:
Count 2}.

Researchers interested in different facets of image analysis, such as
captioning and sentiment analysis, are advised to create and test distinct
prompts tailored to their specific needs. Our focus on object detection is
mostly to showcase Gemini’s capability of solving common computer vision
problems in political science.

### Grading Rubrics

In this subsection, we describe the details of the grading rubrics. When
Gemini does not find any objects or erroneously blocks the processing of an
image, we assign it a one (poor). We also assign it a one when it grossly
misrepresents the image. When Gemini captures some information from the image,
we assign it a two (average). When several objects are correctly identified
but there are noticeable gaps, we assign a three (good). Lastly, when most of
the relevant information is correctly identified, we assign it a four
(excellent). Note that we do not require the object counts to be exactly
correct for a rating of four.

### Data, Materials, and Software Availability

Replication materials will be made available at the Harvard Dataverse,
including all the images, annotations, ratings, as well as the code.

## References

  *   * Grimmer et al. [2021] Grimmer, J., Roberts, M.E., Stewart, B.M.: Machine learning for social science: An agnostic approach. Annual Review of Political Science (2021)
  * Wang [2023] Wang, Y.: On finetuning large language models. Political Analysis (2023)
  * Osnabrügge et al. [2021] Osnabrügge, M., Ash, E., Morelli, M.: Cross-Domain Topic Classification for Political Texts. Political Analysis (2021)
  * Torres and Cantú [2022] Torres, M., Cantú, F.: Learning to see: Convolutional neural networks for the analysis of social science data. Political Analysis (2022)
  * Torres [2023] Torres, M.: A framework for the unsupervised and semi-supervised analysis of visual frames. Political Analysis (2023)
  * Girbau et al. [2023] Girbau, A., Kobayashi, T., Renoust, B., Matsui, Y., Satoh, S.: Face detection, tracking, and classification from large-scale news archives for analysis of key political figures. Political Analysis (2023)
  * Bay and an Luc Van Gool [2006] Bay, H., Luc Van Gool, T.T.: Surf: Speeded up robust features. 9th European Conference on Computer Vision (2006)
  * Arandjelović and Zisserman [2012] Arandjelović, R., Zisserman, A.: Three things everyone should know to improve object retrieval. 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)
  * Girshick et al. [2014] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 Conference on Computer Vision and Pattern Recognition (CVPR) (2014)
  * Gilardi et al. [2023] Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd-workers for text-annotation tasks. Proceedings of the National Academy of Sciences (2023)
  * Huang et al. [2023] Huang, F., Kwak, H., An, J.: Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. WWW ’23 Companion: Companion Proceedings of the ACM Web Conference 2023 (2023)
  * Roberts et al. [2014] Roberts, M.E., Stewart, B.M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S.K., Albertson, B., Rand, D.G.: Structural topic models for open-ended survey responses. American Journal of Political Science (2014)
  * Truhn et al. [2023] Truhn, D., Weber, C.D., Braun, B.J., Bressem, K., Kather, J.N., Kuhl, C., Nebelung, S.: A pilot study on the efficacy of gpt-4 in providing orthopedic treatment recommendations from mri reports. Scientific Reports 13 (2023)