# Three-loop anomalous dimensions
of fixed-charge operators in the SM

###### Abstract

In this Letter we consider renormalization of a class of scalar operators with
fixed hypercharge $Q$ within the Standard Model. We carry out explicit
computation of the corresponding anomalous dimensions up to the three-loop
order. In spite of the fact that our result is gauge-dependent, in the Landau
gauge and in the limit of vanishing weak isospin coupling the expression can
be matched to recent gauge-independent computation based on the large-charge
method. Our result serves an important and non-trivial cross-check of new
developments in large-charge expansion and applications of the latter to
realistic gauge theories. We not only confirm the leading and subleading terms
in perturbative $Q$ expansion up to three loops, but also provide the
expressions for sub-subleading coefficients that at the moment are not
captured by the large-charge approach.

Joint Institute for Nuclear Research, Joliot-Curie, 6, Dubna, 141980, Russia

## Introduction

More than ten years have passed since three-loop renormalization-group (RG)
equations [1, 2, 3] for the full Standard Model (SM) became available in
literature. These results raised immediate interest in scientific community
due to Higgs boson discovery [4, 5] in 2012 and numerous discussions on vacuum
(in)stability of the SM (see, e.g., Refs. [6, 7, 8, 9, 10]).

While during last decade many new results appeared that supplement and go
beyond the three-loop approximation (see, e.g., Refs. [11, 12, 13]), in this
paper we continue the study of the SM at this order. We carry out three-loop
perturbative calculation of the anomalous dimensions for a class of local
operators built from the Higgs field and having fixed weak hypercharge, which
we denote here by $Q$. Such kind of computations play an important role in
matching non-perturbative semiclassical results derived by large-charge
expansion [14, 15, 16, 17, 18, 19, 20, 21, 22, 23] with perturbative
calculations in theories with global symmetries (see, e.g., Refs. [24, 25, 26,
27]).

Our results in the SM are gauge-dependent by construction and, obviously, do
not have direct physical meaning. Nevertheless, recent application of the
large-charge expansion to simple scalar electrodynamics in pioneering paper
[28] demonstrated that perturbative results in the Landau gauge can be matched
via state-operator correspondence to non-perturbative gauge-independent
expressions.

Despite the fact that phenomenological outcome of the large-charge ideas to
realistic gauge theories of particle physics is far from obvious, the result
of matching between non-perturbative (semi-classical) and perturbative
calculations had important consequences to condensed-matter physics,i.e.,
singling out correct gauge-independent order parameter [28]. This motivated us
to go beyond scalar electrodynamics and consider the SM in its full glory.

A natural question arises which symmetry of the SM can be used to define
‘‘fixed-charge’’ configurations and what operators correspond to the lowest-
lying states in this sector. In our current study we does not address the
first issue and refer to recent paper [29] for details and discussions. We
naively consider unbroken SM and compute anomalous dimension of composite
operators built from powers of (electrically) charged would-be goldstone boson
field. The latter being a component of the Higgs doublet also posses weak
hypercharge. Thus, we use powers of the charged would-be goldstone field as a
proxy of operators with fixed weak hypercharge. More elaborate reasoning on
this choice can be found in Ref [29]. It is also important to mention that in
our computation we consider all gauge symmetries of the SM, i.e.,
$SU(3)_{c}\times SU(2)_{W}\times U(1)_{Y}$, and full flavour structure.
However, matching of our expression with large-charge results for non-
vanishing $SU(2)$ coupling is highly non-trivial, if possible at all, since
non-Abelian nature of the latter requires to work with the phase where also
gauge bosons condense [29], which breaks naive correspondence that we silently
assume here.

The paper is organized as follows. In Section 1 we briefly review the SM
Lagrangian with general Yukawa couplings. The definition of operators of our
interest is given in Sec. 2 and some details of calculations are provided. We
present our results and conclusions in Sec. 3.

##  1 The Standard Model

We follow Ref. [30] and consider the Standard Model in the unbroken phase. The
corresponding Lagrangian is given by

${\mathcal{L}}=\mathcal{L}_{\mathrm{gauge}}+\mathcal{L}_{\mathrm{higgs+yukawa}}+\mathcal{L}_{\mathrm{gf}}+\mathcal{L}_{\mathrm{fp}}$
(1)

with

$\displaystyle\mathcal{L}_{\mathrm{gauge}}$
$\displaystyle=-\frac{1}{4}G^{a}_{\mu\nu}G^{a}_{\mu\nu}-\frac{1}{4}W^{i}_{\mu\nu}W^{i}_{\mu\nu}-\frac{1}{4}B_{\mu\nu}B_{\mu\nu}+(D_{\mu}\Phi)^{\dagger}(D_{\mu}\Phi)$
$\displaystyle+i\sum\limits_{j}\left[\bar{Q}_{j}\hat{D}Q_{j}+\bar{L}_{j}\hat{D}L_{j}+\bar{u}_{j}^{R}\hat{D}u_{j}^{R}+\bar{d}_{j}^{R}\hat{D}d_{j}^{R}+\bar{l}_{j}^{R}\hat{D}l_{j}^{R}\right]$
(2)

describing gauge interactions of gluons $G_{\mu}^{a}$, W-bosons $W_{\mu}^{i}$
and a B-boson $B_{\mu}$ with couplings $g_{s}$, $g_{2}$ and $g_{1}$,
respectively. The left-handed quark $Q$ and lepton $L$ fields are $SU(2)_{L}$
doublets. Right-handed up-type $u^{R}$ and down-type $d^{R}$ quarks together
with right-handed charged leptons $l^{R}$ are $SU(2)_{L}$ singlets and do not
couple to W-bosons through covariant derivatives $\hat{D}$. The Higgs doublet
has weak hypercharge $Y_{W}=1$ and can be decomposed as

$\displaystyle\Phi=\left(\begin{array}[]{c}\phi^{+}(x)\\\
\frac{1}{\sqrt{2}}\left(h+i\chi\right)\end{array}\right),\qquad\Phi^{c}=i\sigma^{2}\Phi^{\dagger}=\left(\begin{array}[]{c}\frac{1}{\sqrt{2}}\left(h-i\chi\right)\\\
-\phi^{-}\end{array}\right).$ (7)

with $\Phi^{c}$ being charge-conjugated doublet with $Y_{W}=-1$. The self-
interactions of $\Phi$ and Yukawa interactions of the latter with SM fermions
are described by

$\displaystyle\mathcal{L}_{\mathrm{higgs+yukawa}}$
$\displaystyle=-\lambda\left(\Phi^{\dagger}\Phi\right)^{2}-\sum\limits_{i,j=1,2,3}\left[Y^{ij}_{l}(L^{L}_{i}\Phi)l^{R}_{j}+\mathrm{h.c.}\right]$
$\displaystyle-\sum\limits_{i,j=1,2,3}\left[Y^{ij}_{u}(Q^{L}_{i}\Phi^{c})u^{R}_{j}+Y^{ij}_{d}(Q^{L}_{i}\Phi)d^{R}_{j}+\mathrm{h.c.}\right],$
(8)

where we emphasize the flavour structure of the SM described by matrix Yukawa
couplings111In this computation we neglect possible contribution from right-
handed neutrinos that can couple to the Higgs boson via the corresponding
Yukawa coupling $Y_{\nu}$. $Y_{u}$, $Y_{d}$, and $Y_{l}$. We omit the Higgs-
boson mass parameter, since it does not affect our computation. As for the
gauge-fixing $\mathcal{L}_{\mathrm{gf}}$ and ghost terms
$\mathcal{L}_{\mathrm{fp}}$, we use here simple linear $R_{\xi}$ gauge:

$\displaystyle\mathcal{L}_{\mathrm{gf}}$
$\displaystyle=-\frac{1}{2\xi_{\alpha}}G_{\alpha}G_{\alpha},$
$\displaystyle\mathcal{L}_{\mathrm{fp}}$
$\displaystyle=-\bar{c}_{\alpha}\frac{\delta
G_{\alpha}}{\delta\theta^{\beta}}c_{\beta}$ (9)

where $\alpha,\beta=(G,W,B)$, , and $\delta G_{\alpha}/\delta\theta^{\beta}$
is the variation of the gauge-fixing function
$G_{\alpha}=\partial_{\mu}A^{\alpha}_{\mu}$
[$A_{\mu}^{\alpha}=(G_{\mu}^{a},W_{\mu}^{i},B_{\mu})$] under infinitesimal
gauge transformations with parameters $\theta_{\beta}$ corresponding to
$SU(3)_{c}\times SU(2)_{L}\times U(1)_{Y}$ local symmetries.

##  2 Fixed-charge operators

Let us now consider a family of local operators that have fixed hypercharge.
We now denote it as $Q$ to be in contact with literature on large-charge
expansion. We naively trasfer our experience in theories with global [27] and
local symmetries [28] and study the renormalization of $\mathcal{O}_{Q}$ built
from would-be goldstone boson fields $\phi^{+}$

$\displaystyle\mathcal{O}_{Q}=(\phi^{+})^{Q}\equiv\underbrace{\Phi^{1}\ldots\Phi^{1}}_{Q},$
(10)

where we indicate that the considered operators at fixed $Q$ belong to
$(Q+1)$-plet that transforms under $SU(2)_{L}$. Due to this symmetry, all
components of the multiplets renormalize in the same way and have the same
anomalous dimensions. In what follows we silently assume dimensional
regularization and define all the renormalization constants in the (modified)
minimal-subtraction scheme ${\overline{\mathrm{MS}}}$.

One can use simple combinatorial argument and demonstrate that the anomalous
dimension $\gamma_{Q}$ of operators (10) for general $Q$ can be written in
perturbation theory as

$\displaystyle\gamma_{Q}=\sum\limits_{l=1}^{\infty}\gamma_{Q}^{(l-\mathrm{loop})},\qquad\gamma_{Q}^{(l-\mathrm{loop})}\equiv\sum\limits_{k=0}^{l}C_{lk}Q^{l+1-k},$
(11)

where the coefficients $C_{lk}$ depend on the SM couplings and gauge-fixing
parameters. One sees that $l$-loop contribution is a polynomial of degree
$l+1$ in $Q$. As a consequence, at three loops we can recover full dependence
of $\gamma^{(3)}_{Q}$ on $Q$ by carrying out explicit computations for fixed
$Q=1,2,3,4$.

The case $Q=1$ corresponds to the Higgs field itself. The renormalization of
the latter is known from literature and it is not a problem to derive the
anomalous dimension from the relation between bare $\Phi_{0}$ and renormalized
$\Phi$ fields:

$\displaystyle\Phi_{0}=Z^{1/2}_{\Phi}\Phi,\qquad\gamma_{\Phi}\equiv\frac{dZ_{\Phi}}{d\ln\mu}\cdot
Z^{-1}_{\Phi}=\gamma_{Q=1}.$ (12)

Here $\mu$ denotes the renormalization scale. The cases $Q=2,3,4$ were
considered separately. We computed three-loop Green functions with
$\mathcal{O}_{Q}$ insertions and utilized the relation

$\displaystyle[\mathcal{O}_{Q}]_{R}=Z_{Q}[\mathcal{O}_{Q}],\qquad\gamma_{Q}\equiv-\frac{dZ_{Q}}{d\ln\mu}\cdot
Z^{-1}_{Q}$ (13)

to derive the corresponding anomalous dimensions $\gamma_{Q=2,3,4}$. Let us
mention that we heavily rely on the computer setup that we used to compute
three-loop RG functions in the SM [3].

##  3 Results and Conclusions

Having at hand $\gamma_{Q}$ for fixed $Q=1,2,3,4$, one can derive the
expression for all the coefficients $C_{lk}$ up to $l=3$. In what follows we
present our result in the Landau gauge $\xi_{\alpha}=0$ and use the following
notation for gauge couplings and Higgs self-interaction

$\displaystyle a_{s}=\frac{g_{s}^{2}}{16\pi^{2}},\quad
a_{1}=\frac{g_{1}^{2}}{16\pi^{2}},\quad
a_{2}=\frac{g_{2}^{2}}{16\pi^{2}},\quad
a_{\lambda}=\frac{\lambda}{16\pi^{2}}.$ (14)

For the traces of Yukawa matrices that appear in the result we introduce the
abbreviations

$\displaystyle\mathcal{Y}_{d}$
$\displaystyle=\frac{\operatorname{Tr}Y_{f}Y_{f}^{\dagger}}{16\pi^{2}},\quad\mathcal{Y}_{ff^{\prime}}=\frac{\operatorname{Tr}Y_{f}Y_{f}^{\dagger}Y_{f^{\prime}}Y_{f^{\prime}}^{\dagger}}{(16\pi^{2})^{2}},$
$\displaystyle\mathcal{Y}_{ff^{\prime}f^{\prime\prime}}$
$\displaystyle=\frac{\operatorname{Tr}Y_{f}Y_{f}^{\dagger}Y_{f^{\prime}}Y_{f^{\prime}}^{\dagger}Y_{f^{\prime\prime}}Y_{f^{\prime\prime}}^{\dagger}}{(16\pi^{2})^{3}},\quad
f,f^{\prime},f^{\prime\prime}\in\\{u,d,l\\}.$ (15)

The leading coefficients in $Q$ expansion up to three loops are given
by222Note that we should rescale $\lambda\to\lambda/6$ to make direct
comparison with Ref. [29].

$\displaystyle C_{10}$ $\displaystyle=2a_{\lambda},$ $\displaystyle C_{20}$
$\displaystyle=-8a_{\lambda}^{2},$ $\displaystyle C_{30}$
$\displaystyle=64a_{\lambda}^{3}$ (16)

and coincide with those of $\phi^{4}$ theory. The expressions for subleading
coefficients take the form

$\displaystyle C_{11}=$
$\displaystyle-\frac{3a_{1}}{4}-2a_{\lambda}+3\mathcal{Y}_{d}+3\mathcal{Y}_{u}+\mathcal{Y}_{l}{\color[rgb]{0,0,1}-\frac{9a_{2}}{4}},$
(17) $\displaystyle C_{21}=$
$\displaystyle\frac{a_{1}^{2}}{16}-a_{1}\left(2a_{\lambda}-{\color[rgb]{0,0,1}\frac{a_{2}}{8}}\right)+{\color[rgb]{0,0,1}\frac{3a_{2}^{2}}{16}}$
$\displaystyle+4a_{\lambda}^{2}-4a_{\lambda}(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})+2(3\mathcal{Y}_{uu}+3\mathcal{Y}_{dd}+\mathcal{Y}_{ll}),$
(18) $\displaystyle C_{31}=$
$\displaystyle-\frac{1}{16}\left(a_{1}^{3}+{\color[rgb]{0,0,1}3a_{1}^{2}a_{2}+3a_{1}a_{2}^{2}+3a_{2}^{3}}\right)\left(1-9\zeta_{3}\right)-a_{\lambda}\left[a_{1}(a_{1}+{\color[rgb]{0,0,1}2a_{2}})(1+3\zeta_{3})\right]$
$\displaystyle+12a_{\lambda}^{2}a_{1}(3-2\zeta_{3})-32a_{\lambda}^{3}(8-9\zeta_{3})+16a_{\lambda}^{2}(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})$
$\displaystyle-8a_{\lambda}(3\mathcal{Y}_{uu}+3\mathcal{Y}_{dd}+\mathcal{Y}_{ll})(1-3\zeta_{3})-8\zeta_{3}\left(3\mathcal{Y}_{uuu}+3\mathcal{Y}_{ddd}+\mathcal{Y}_{lll}\right)$
$\displaystyle+{\color[rgb]{0,0,1}4a_{\lambda}^{2}a_{2}(13-12\zeta_{3})}-{\color[rgb]{0,0,1}a_{\lambda}a_{2}^{2}(5+12\zeta_{3})},$
(19)

where we highlight terms proportional to the $SU(2)_{L}$ coupling $g_{2}$.
Again contributions involving only $a_{\lambda}$ reproduce the expressions in
$O(4)$-symmetric $\phi^{4}$ model.

All the above-mentioned coefficients can be compared to all-order results [29]
in the limit $g_{2}=0$. We go beyond this approximation and provide also
subsubleading terms. At two loops we have

$\displaystyle C_{22}$
$\displaystyle=\frac{425a_{1}^{2}}{96}-{\color[rgb]{0,0,1}\frac{277a_{2}^{2}}{32}}+a_{1}\left[2a_{\lambda}+\frac{5}{24}(5\mathcal{Y}_{d}+15\mathcal{Y}_{l}+17\mathcal{Y}_{u})+{\color[rgb]{0,0,1}\frac{7a_{2}}{16}}\right]$
$\displaystyle+{\color[rgb]{0,0,1}\frac{15}{8}a_{2}(3\mathcal{Y}_{d}+\mathcal{Y}_{l}+3\mathcal{Y}_{u})}+10a_{\lambda}^{2}+4a_{\lambda}(3\mathcal{Y}_{d}+\mathcal{Y}_{l}+3\mathcal{Y}_{u})$
$\displaystyle+20a_{s}(\mathcal{Y}_{d}+\mathcal{Y}_{u})-\frac{1}{4}(51\mathcal{Y}_{dd}+51\mathcal{Y}_{uu}+17\mathcal{Y}_{ll}-6\mathcal{Y}_{ud}).$
(20)

At three loops we extract two coefficients that are not predicted by semi-
classical approach. To save space333Full results in the linear $R_{\xi}$ gauge
are available as supplementary Mathematica files. we present them in the limit
$g_{2}=0$:

$\displaystyle C_{32}$
$\displaystyle=\frac{9}{16}a_{1}^{3}(14-3\zeta_{3})+\frac{3}{32}a_{\lambda}a_{1}^{2}(16\zeta_{3}-267)-4a_{1}a_{\lambda}^{2}(32-33\zeta_{3})$
$\displaystyle-\frac{a_{1}^{2}}{6}\left[\mathcal{Y}_{d}(13+3\zeta_{3})-\mathcal{Y}_{l}(69-81\zeta_{3})-\mathcal{Y}_{u}(53-75\zeta_{3})\right]$
$\displaystyle+\frac{a_{1}a_{\lambda}}{8}\left[\mathcal{Y}_{d}(179+224\zeta_{3})-\mathcal{Y}_{l}(167-288\zeta_{3})-\mathcal{Y}_{u}(25-416\zeta_{3})\right]$
$\displaystyle+\frac{a_{1}}{6}\left[\mathcal{Y}_{dd}(51\zeta_{3}-146)+\mathcal{Y}_{ll}(18-63\zeta_{3})-\mathcal{Y}_{uu}(86+21\zeta_{3})\right]$
$\displaystyle+2a_{\lambda}^{3}(353-432\zeta_{3})-8a_{\lambda}^{2}(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})$
$\displaystyle-12a_{\lambda}a_{s}(17-16\zeta_{3})(\mathcal{Y}_{d}+\mathcal{Y}_{u})+16a_{s}(5-6\zeta_{3})(\mathcal{Y}_{dd}+\mathcal{Y}_{uu})$
$\displaystyle-6a_{\lambda}\left[9\mathcal{Y}_{d}^{2}+9\mathcal{Y}_{u}^{2}+\mathcal{Y}_{l}^{2}+6(\mathcal{Y}_{u}+\mathcal{Y}_{d})\mathcal{Y}_{l}+18\mathcal{Y}_{d}\mathcal{Y}_{u}\right]$
$\displaystyle+\frac{a_{\lambda}}{4}\left[(3\mathcal{Y}_{uu}+3\mathcal{Y}_{dd}+\mathcal{Y}_{ll})(235-240\zeta_{3})+\mathcal{Y}_{ud}(510-576\zeta_{3})\right]$
$\displaystyle+4\left[9(\mathcal{Y}_{d}+\mathcal{Y}_{u})(\mathcal{Y}_{dd}+\mathcal{Y}_{uu})+3(\mathcal{Y}_{d}+\mathcal{Y}_{u})\mathcal{Y}_{ll}+3(\mathcal{Y}_{dd}+\mathcal{Y}_{uu})\mathcal{Y}_{l}+\mathcal{Y}_{l}\mathcal{Y}_{ll}\right]$
$\displaystyle+6(\mathcal{Y}_{udd}+\mathcal{Y}_{uud})-2(3\mathcal{Y}_{ddd}+3\mathcal{Y}_{uuu}+\mathcal{Y}_{lll})(11-12\zeta_{3}),$
(21) $\displaystyle C_{33}$
$\displaystyle=\frac{a_{1}^{3}}{432}(23678-13275\zeta_{3})+\frac{a_{1}^{2}a_{\lambda}}{32}(911-96\zeta_{3})+a_{1}a_{\lambda}^{2}(107-108\zeta_{3})$
$\displaystyle-\frac{a_{1}^{2}}{3456}\left[174799\mathcal{Y}_{u}+20311\mathcal{Y}_{d}+200637\mathcal{Y}_{l}-288\zeta_{3}(151\mathcal{Y}_{u}-23\mathcal{Y}_{d}+279\mathcal{Y}_{l})\right]$
$\displaystyle+\frac{a_{1}a_{\lambda}}{8}[25\mathcal{Y}_{u}-179\mathcal{Y}_{d}+167\mathcal{Y}_{l}-32\zeta_{3}(7\mathcal{Y}_{d}+9\mathcal{Y}_{l}+13\mathcal{Y}_{u})]$
$\displaystyle+\frac{11}{4}a_{1}^{2}a_{s}(15-16\zeta_{3})+\frac{a_{1}a_{s}}{36}[\mathcal{Y}_{d}(720\zeta_{3}-991)+\mathcal{Y}_{u}(2448\zeta_{3}-2419)]$
$\displaystyle+\left(6a_{\lambda}-\frac{3a_{1}}{4}\right)(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})^{2}-\frac{61a_{\lambda}^{2}}{2}(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})$
$\displaystyle+\frac{a_{1}}{48}[55\mathcal{Y}_{uu}+259\mathcal{Y}_{dd}-243\mathcal{Y}_{ll}-186\mathcal{Y}_{ud}+24\zeta_{3}(\mathcal{Y}_{uu}+\mathcal{Y}_{dd}+3\mathcal{Y}_{ll}+16\mathcal{Y}_{ud})]$
$\displaystyle+\frac{a_{\lambda}}{4}(144\zeta_{3}(3\mathcal{Y}_{uu}+3\mathcal{Y}_{dd}+\mathcal{Y}_{ll}+4\mathcal{Y}_{ud})-143(3\mathcal{Y}_{uu}+3\mathcal{Y}_{dd}+\mathcal{Y}_{ll})-510\mathcal{Y}_{ud})$
$\displaystyle+\frac{2a_{s}^{2}}{3}(311-36\zeta_{3})(\mathcal{Y}_{d}+\mathcal{Y}_{u})-12a_{\lambda}a_{s}(16\zeta_{3}-17)(\mathcal{Y}_{d}+\mathcal{Y}_{u})$
$\displaystyle-\frac{a_{s}}{2}[145\mathcal{Y}_{uu}+145\mathcal{Y}_{dd}-114\mathcal{Y}_{ud}-48\zeta_{3}(\mathcal{Y}_{uu}+\mathcal{Y}_{dd}-2\mathcal{Y}_{ud})]$
$\displaystyle+\frac{1}{2}(12\mathcal{Y}_{uu}+12\mathcal{Y}_{dd}+4\mathcal{Y}_{ll}-3\mathcal{Y}_{ud})(3\mathcal{Y}_{u}+3\mathcal{Y}_{d}+\mathcal{Y}_{l})-2a_{\lambda}^{3}(275-288\zeta_{3})$
$\displaystyle+\frac{1}{16}\left[(327-208\zeta_{3})(3\mathcal{Y}_{uuu}+3\mathcal{Y}_{ddd}+\mathcal{Y}_{lll})-57(\mathcal{Y}_{udd}+\mathcal{Y}_{uud})\right].$
(22)

Let us make a few comments on the obtained result. First of all, we stress
that it is _not_ gauge-independent. Secondly, one can notice that there is no
dependence on the strong coupling $a_{s}$ in first two leading coefficient
$C_{l0}$ and $C_{l1}$, meaning that strong interactions play a role of
‘‘spectator’’ when subsubleading terms are ignored. Finally, $C_{l0}$ and
$C_{l1}$, when evaluated in the Landau gauge and in the limit of vanishing
$g_{2}$, can be compared to large-charge all-order predictions of recent study
[29]. Perfect agreement that was found serves as a welcome cross-check of both
computations.

It is also worth mentioning that the anomalous dimensions obtained in Ref.
[29] are gauge-independent by construction. As a consequence, they should
correspond to some lowest-lying gauge-independent operators with fixed
hypercharge and can not be directly connected to our current result based on
Eq. (10). Nevertheless, when non-Abelian interactions are ignored, one can
follow the reasoning of [28] and introduce a generalization of our operators
$\mathcal{O}_{Q}$ that accounts for dressing of the charge with coherent state
of Abelian gauge bosons describing Coulomb-like field. In the Landau gauge
these gauge-independent operators can be reduced to $\mathcal{O}_{Q}$ given in
Eq. (10), thus, explaining the coincidence.

For non-Abelian symmetries the trick seems not to be applicable in the naive
form, and the case $g_{2}\neq 0$ requires further investigation [29]. However,
we believe that our present result can shed light on this issue. We also note
that the Higgs boson observed at the LHC, although being electrically neutral,
also posses weak hypercharge. As a consequence, the properties of operators
with large hypercharge can be important, e.g., in connection with multiple
higgs production (see, e.g., recent discussion [31] and references therein).

## Acknowledgments

We appreciate fruitful discussions with Oleg Antipin on the application of
large-charge expansion in gauge theories. We also grateful to Francesco
Sannino for his comments on the manuscript and thank all the authors of Ref.
[29] for sharing with us the preliminary results of their inspiring paper.

## References

  * [1] _Mihaila L.N., Salomon J., Steinhauser M._ Gauge Coupling Beta Functions in the Standard Model to Three Loops // Phys.Rev.Lett. —- 2012\. —- V. 108. —- P. 151602. —- arXiv:1201.5868 [hep-ph].
  * [2] _Bednyakov A.V., Pikelner A.F., Velizhanin V.N._ Three-loop SM beta-functions for matrix Yukawa couplings // Phys. Lett.B. —- 2014\. —- V. 737. —- P. 129–134. —- arXiv:1406.7171 [hep-ph].
  * [3] _Bednyakov A.V., Pikelner A.F., Velizhanin V.N._ Three-loop Higgs self-coupling beta-function in the Standard Model with complex Yukawa matrices // Nucl. Phys.B. —- 2014\. —- V. 879. —- P. 256–267. —- arXiv:1310.3806 [hep-ph].
  * [4] _Aad G. et al._ [ATLAS Collaboration] Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC // Phys. Lett.B. —- 2012\. —- V. 716. —- P. 1–29. —- arXiv:1207.7214 [hep-ex].
  * [5] _Chatrchyan S. et al._ [CMS Collaboration] Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC // Phys. Lett.B. —- 2012\. —- V. 716. —- P. 30–61. —- arXiv:1207.7235 [hep-ex].
  * [6] _Bezrukov F., Kalmykov M.Y., Kniehl B.A., Shaposhnikov M._ Higgs Boson Mass and New Physics // JHEP. —- 2012\. —- V. 10. —- P. 140. —- arXiv:1205.2893 [hep-ph].
  * [7] _Degrassi G., Di Vita S., Elias-Miro J., Espinosa J.R., Giudice G.F., Isidori G., Strumia A._ Higgs mass and vacuum stability in the Standard Model at NNLO // JHEP. —- 2012\. —- V. 08. —- P. 098. —- arXiv:1205.6497 [hep-ph].
  * [8] _Buttazzo D., Degrassi G., Giardino P.P., Giudice G.F., Sala F., Salvio A., Strumia A._ Investigating the near-criticality of the Higgs boson // JHEP. —- 2013\. —- V. 12. —- P. 089. —- arXiv:1307.3536 [hep-ph].
  * [9] _Antipin O., Gillioz M., Krog J., Mølgaard E., Sannino F._ Standard Model Vacuum Stability and Weyl Consistency Conditions // JHEP. —- 2013\. —- V. 08. —- P. 034. —- arXiv:1306.3234 [hep-ph].
  * [10] _Bednyakov A.V., Kniehl B.A., Pikelner A.F., Veretin O.L._ Stability of the Electroweak Vacuum: Gauge Independence and Advanced Precision // Phys. Rev. Lett. —- 2015\. —- V. 115, no. 20. —- P. 201802. —- arXiv:1507.08833.
  * [11] _Bednyakov A.V., Pikelner A.F._ Four-loop strong coupling beta-function in the Standard Model // Phys. Lett.B. —- 2016\. —- V. 762. —- P. 151–156. —- arXiv:1508.02680.
  * [12] _Chetyrkin K.G., Zoller M.F._ Leading QCD-induced four-loop contributions to the $\beta$-function of the Higgs self-coupling in the SM and vacuum stability // JHEP. —- 2016\. —- V. 06. —- P. 175. —- arXiv:1604.00853.
  * [13] _Davies J., Herren F., Poole C., Steinhauser M., Thomsen A.E._ Gauge Coupling $\beta$ Functions to Four-Loop Order in the Standard Model // Phys. Rev. Lett. —- 2020\. —- V. 124, no. 7. —- P. 071803. —- arXiv:1912.07624.
  * [14] _Hellerman S., Orlando D., Reffert S., Watanabe M._ On the CFT Operator Spectrum at Large Global Charge // JHEP. —- 2015\. —- V. 12. —- P. 071. —- arXiv:1505.01537.
  * [15] _Monin A., Pirtskhalava D., Rattazzi R., Seibold F.K._ Semiclassics, Goldstone Bosons and CFT data // JHEP. —- 2017\. —- V. 06. —- P. 011. —- arXiv:1611.02912.
  * [16] _Alvarez-Gaume L., Loukas O., Orlando D., Reffert S._ Compensating strong coupling with large charge // JHEP. —- 2017\. —- V. 04. —- P. 059. —- arXiv:1610.04495.
  * [17] _Banerjee D., Chandrasekharan S., Orlando D._ Conformal dimensions via large charge expansion // Phys. Rev. Lett. —- 2018\. —- V. 120, no. 6. —- P. 061603. —- arXiv:1707.00711.
  * [18] _Badel G., Cuomo G., Monin A., Rattazzi R._ The Epsilon Expansion Meets Semiclassics // JHEP. —- 2019\. —- V. 11. —- P. 110. —- arXiv:1909.01269.
  * [19] _Orlando D., Reffert S., Sannino F._ A safe CFT at large charge // JHEP. —- 2019\. —- V. 08. —- P. 164. —- arXiv:1905.00026.
  * [20] _Gaumé L.A., Orlando D., Reffert S._ Selected topics in the large quantum number expansion // Phys. Rept. —- 2021\. —- V. 933. —- P. 1–66. —- arXiv:2008.03308.
  * [21] _Antipin O., Bersini J., Sannino F., Wang Z.W., Zhang C._ Charging the $O(N)$ model // Phys. Rev. D. —- 2020\. —- V. 102, no. 4. —- P. 045011. —- arXiv:2003.13121.
  * [22] _Giombi S., Helfenberger E., Khanchandani H._ Long range, large charge, large N // JHEP. —- 2023\. —- V. 01. —- P. 166. —- arXiv:2205.00500.
  * [23] _Antipin O., Bersini J., Panopoulos P._ Yukawa interactions at large charge // JHEP. —- 2022\. —- V. 10. —- P. 183. —- arXiv:2208.05839.
  * [24] _Jack I., Jones D.R.T._ Anomalous dimensions at large charge in d=4 O(N) theory // Phys. Rev. D. —- 2021\. —- V. 103, no. 8. —- P. 085013. —- arXiv:2101.09820.
  * [25] _Jack I., Jones D.R.T._ Anomalous dimensions at large charge for U(N)×U(N) theory in three and four dimensions // Phys. Rev. D. —- 2021\. —- V. 104, no. 10. —- P. 105017. —- arXiv:2108.11161.
  * [26] _Jin Q., Li Y._ Five-loop anomalous dimensions of $\phi$Q operators in a scalar theory with O(N) symmetry // JHEP. —- 2022\. —- V. 10. —- P. 084. —- arXiv:2205.02535.
  * [27] _Bednyakov A., Pikelner A._ Six-loop anomalous dimension of the $\phi^{Q}$ operator in the $O(N)$ symmetric model // Phys. Rev. D. —- 2022\. —- V. 106, no. 7. —- P. 076015. —- arXiv:2208.04612.
  * [28] _Antipin O., Bednyakov A., Bersini J., Panopoulos P., Pikelner A._ Gauge Invariance at Large Charge // Phys. Rev. Lett. —- 2023\. —- V. 130, no. 2. —- P. 021602. —- arXiv:2210.10685.
  * [29] _Antipin O., Bersini J., Panopoulos P., Sannino F., Wang Z.W._ Infinite order results for charged sectors of the Standard Model. —- 2023\. —- 12\. —- arXiv:2312.12963.
  * [30] _Bednyakov A., Pikelner A., Velizhanin V._ Anomalous dimensions of gauge fields and gauge coupling beta-functions in the Standard Model at three loops // JHEP. —- 2013\. —- V. 1301. —- P. 017. —- arXiv:1210.6873 [hep-ph].
  * [31] _Demidov S.V., Farkhtdinov B.R., Levkov D.G._ Suppression exponent for multiparticle production in $\lambda$$\phi$4 theory // JHEP. —- 2023\. —- V. 02. —- P. 205. —- arXiv:2212.03268.


# Towards Coupling Full-disk and Active Region-based Flare Prediction for
Operational Space Weather Forecasting

Chetraj Pandey 1,∗, Anli Ji 1, Rafal A. Angryk 1, Manolis K. Georgoulis 2, and
Berkay Aydin 1

###### Abstract

Solar flare prediction is a central problem in space weather forecasting and
has captivated the attention of a wide spectrum of researchers due to recent
advances in both remote sensing as well as machine learning and deep learning
approaches. The experimental findings based on both machine and deep learning
models reveal significant performance improvements for task specific datasets.
Along with building models, the practice of deploying such models to
production environments under operational settings is a more complex and often
time-consuming process which is often not addressed directly in research
settings. We present a set of new heuristic approaches to train and deploy an
operational solar flare prediction system for $\geq$M1.0-class flares with two
prediction modes: full-disk and active region-based. In full-disk mode,
predictions are performed on full-disk line-of-sight magnetograms using deep
learning models whereas in active region-based models, predictions are issued
for each active region individually using multivariate time series data
instances. The outputs from individual active region forecasts and full-disk
predictors are combined to a final full-disk prediction result with a meta-
model. We utilized an equal weighted average ensemble of two base learners’
flare probabilities as our baseline meta learner and improved the capabilities
of our two base learners by training a logistic regression model. The major
findings of this study are: (i) We successfully coupled two heterogeneous
flare prediction models trained with different datasets and model architecture
to predict a full-disk flare probability for next 24 hours, (ii) Our proposed
ensembling model, i.e., logistic regression, improves on the predictive
performance of two base learners and the baseline meta learner measured in
terms of two widely used metrics True Skill Statistic (TSS) and Heidke Skill
Score (HSS), and (iii) Our result analysis suggests that the logistic
regression-based ensemble (Meta-FP) improves on the full-disk model (base
learner) by $\sim$9% in terms TSS and $\sim$10% in terms of HSS. Similarly, it
improves on the AR-based model (base learner) by $\sim$17% and $\sim$20% in
terms of TSS and HSS respectively. Finally, when compared to the baseline meta
model, it improves on TSS by $\sim$10% and HSS by $\sim$15%.

##  1 Keywords:

solar flares, solar magnetograms, ensemble, machine learning, deep learning

##  2 Introduction

A solar flare is an intense burst of electromagnetic radiation through
magnetic reconnection and plasma instability coming from the release of
magnetic energy associated with active regions (AR) and they transpire as a
sudden brightening of light on the Sun’s corona Toriumi and Wang (2019).
Coronal mass ejections (CMEs), which are often associated with solar flares,
have comparable energies, and can release large amounts of mass resulting into
major geomagnetic storms which creates intense currents in the Earth’s
magnetosphere, changes in the radiation belts, and in the ionosphere Feng et
al. (2020). When particles emitted by the Sun are accelerated during a flare
or by a CME event and reach the Earth along interplanetary magnetic field
lines, Solar energetic particle (SEP) events are produced Núñez and Paul-Pena
(2020). Primarily, solar flares are considered to be the central phenomena in
space weather forecasting, and this paper discusses on the predictive models
for solar flares. Solar flares can induce intense variation in Earth’s
magnetic field, causing potential disruptions to many stakeholders such as the
electricity supply chain, airlines industry, astronauts in space, and
communication systems including satellites and radio. Forecasting solar flares
has been a major challenge in heliophysics owing to the yet unsolved
fundamental cause of this phenomenon which makes it difficult to predict the
exact occurrence of a flare, especially for relatively large ones. However,
recent advancements in machine learning and deep learning methods have
demonstrated great experimental success and catalyzed the efforts in
prediction of solar flares, which captivated the interest of many
interdisciplinary researchers Li et al. (2020); Nishizuka et al. (2018); Huang
et al. (2018). Developing predictive models for flare prediction is limited to
the nature, quantity, and quality of flaring instances as well as the
inductive bias of learning algorithms when predicting such flare events. As a
consequence of the intrinsic limitations pre-incorporated by the predictive
models during problem formulation or model selection or utilizing different
data products, an individual flare prediction model is limited in performance.
Although all the models built so far for flare forecasting have limitations,
different comprehensions and insights on data distribution are still valuable
for making the final decision in an operational flare forecasting system.
Therefore, it is intuitive to use as many pieces of information that can be
gathered from different sets of models such as machine learning or deep
learning models obtained from different data modalities in terms of active
region magnetogram patches, full-disk magnetograms or magnetogram’s metadata
(magnetic field parameters) to issue a reduced risk prediction.

In active region-based models, predictions are issued for certain areas on the
Sun with greatly enhanced magnetic flux, known as active regions. Active
regions have lifetimes of days to month, feature strong and entangled magnetic
fields and are the exclusive locations of strong flares and major eruptions,
including fast coronal mass ejections (CMEs). This said, only a slim minority
(10$\%$ or less) of active regions appearing in a given solar cycle provide
flares of GOES class $\geq$M1.0 and fast CMEs (e.g., Georgoulis et al. (2019);
Toriumi and Wang (2019)). These regions can host solar eruptions. To employ
active region-based models in an operational setting, individual active region
forecasts are aggregated by computing the probability of flare from at least
one active region assuming conditional independence and then these flare
probabilities are used to compute a full-disk flare occurrence probability.
However, for an operational system, working with near-real time data and
issuing near-real time predictions, active region-based models relying on
magnetic field observations possess a limited forecasting ability as they
restrict the training datasets within central regions ($\pm$70∘) due to severe
projection effects Hoeksema et al. (2014). Besides the unreliable
measurements, foreshortening closer to the solar limbs greatly impacts the
operational use of magnetic field data. This leads to reduction in significant
information required to make reliable flare predictions in active regions.
Moreover, predictions from active region-based models often rely on sampled
subset of statistical features that were used to train the model and therefore
when examining forecasts from different subsets of features, it is common to
observe that for the similar condition of the photospheric magnetic field,
they can give varying values for prediction probabilities of a particular
flare to happen.

To account for the limitations of active region-based flare predictors, full-
disk prediction models provide a complementary approach for operational flare
forecasting systems Pandey et al. (2021). The full-disk model utilize the
compressed line-of-sight magnetograms and these magnetograms are used for
shape based parameters (such as size, directionality, borders of sunspots) and
do not possess the magnetic field properties as in the magnetogram rasters
which is advantageous over the active region-based models where individual
active region magnetic field parameters used near the limb are more prone to
projection effects. The significant part of an operational flare forecasting
model is to issue a reliable forecast for which we use a heterogeneous
ensemble that combines two different base learners. In addition, to address
the operational aspect of our system, we consider two essential system-level
criteria: (i) near-real-time availability of input data is ensured given that
both of our base learners are trained with line-of-sight magnetograms and
physical parameters obtained from a line-of-sight magnetograms and vector
magnetograms available at a cadence of 12 minutes, and (ii) our proposed
system is scalable in a sense that it allows the flexibility of adding a new
base learner (if needed in the future) in the system as it will be one step
away from retraining the ensemble and deploying it back to our forecasting
system.

In this work, to issue more reliable forecasts in an operational settings, we
propose a heuristic ensemble approach which consolidates the predictive
results of the two aforementioned prediction modalities into one combined
solar flare forecast. The major contributions of this paper are following: we
present a methodology on how to train and validate an ensemble flare
prediction model in regard to its operations-ready characteristics. The
ensemble combines the predictions from two base learners: (i) a deep learning-
based full-disk flare predictor using SDO/HMI images and (ii) a set of
probabilistic predictions from a time series classifier utilizing active
region patches’ magnetic field metadata in the form of multivariate time
series. For both base learners, we use the similar time-segmented tri-monthly
data partitioning strategy Pandey et al. (2021) to perform 3-fold cross-
validation experiments. Finally, we use the probability scores of these two
base learners obtained from the validation and test partitions to train and
validate our proposed meta-learner which converges to a more robust full-disk
flare predictor.

The remainder of this paper is organized as follows. In Sec. 3, we present the
related work on ensemble solar flare forecasting models. In Sec. 4, we provide
a detailed workflow of our methodology. In Sec. 5, we present our detailed
experimental evaluation with settings and results. In Sec. 6 we present a
discussion on the ensembles created and, lastly, in Sec. 7, we present our
conclusions and discuss future work.

##  3 Related Works

The idea of automatically extracting forecast patterns from the large volume
of intrinsic magnetic field data on the photosphere of the sun using machine
learning methods has begun from the early 1990’s Aso et al. (1994). Since
then, with the rapid development in machine learning and deep learning
approaches, a number of research groups Nishizuka et al. (2018); Huang et al.
(2018); Li et al. (2020), Nishizuka et al. (2021), and references therein
present their efforts in applying such methods to build ﬂare forecasting
models.

In recent years, Li et al. (2020); Huang et al. (2018) used a deep learning
model based on CNN with different data products for flare forecasting.
Although they show an impressive performance on flare classification, they
limit the scope of the prediction to smaller areas by using active region-
based data within $\pm$30∘ to 45∘ of the central meridian of the Sun which may
counter their performance for true operational forecasting. In addition,
Florios et al. (2018) calculated physical features of flaring and non-flaring
ARs obtained from the SDO/HMI’s near-real-time vector magnetogram data and
trained SVMs, multilayer perceptrons (MLPs), and decision tree algorithms to
predict occurrences of $\geq$M1.0-class and $\geq$C1.0-class flares with a
forecast horizon of 24 hours. In Benvenuto et al. (2018), a combination of
supervised lasso regression for identifying the significant features and then
an unsupervised fuzzy clustering is used for the classification of
$\geq$M1.0-class and $\geq$C1.0-class flares. Furthermore, Park et al. (2018);
Pandey et al. (2021) uses full-disk magnetograms data as a point in time
observation with CNN based deep learning models, which have limitations in
capturing the evolution of solar flares and they do not account for flares
that are on the eastern-limb of the Sun. Overall, some methods are appropriate
for constructing prediction models for the temporal data variation, whereas
others are beneficial for spatial data variation, which demands a need for a
coupled hybrid model that can exploit the gains of multiple models.

Jonas et al. (2018) designed a time series data set using photospheric and
coronal images from HMI/SDO and AIA/SDO instruments to forecast
$\geq$M1.0-class flares within the next 24 h. They utilize random partitioning
of datasets into 80$\%$ and 20$\%$ for training and testing the linear
classifier. Apart from devising flare forecasting as a binary classification
task, Abduallah et al. (2021) formulates it as a multiclass classification
problem to classify B-, C-, M- and X-class flares by utilizing the physical
parameters within $\pm$70∘ provided by the SHARP series of HMI/SDO. Finally,
the author uses majority voting as an ensemble to issue a final flare forecast
from three different models trained on the same data. The training procedure
in their work uses random 10-fold cross-validation.

Instead of using a single prediction model, ensembles use a set of predictions
and combine these results to improve on a single-model prediction. In
addition, an ensemble can be created with a single model itself by perturbing
its initial conditions or parameter settings to produce multiple results and
then combine those results into one called homogeneous ensembles Breiman
(1996); Freund and Schapire (1996). Flare forecasting problems also make use
of decision tree-based homogeneous ensembles. Liu et al. (2017a) apply random
forest (RF) Breiman (2001) – a meta-algorithm that fits a number of decision
tree classifiers on different sub-samples of a dataset and utilizes averaging
to improve the model’s performance. Similarly, Nishizuka et al. (2017)
employed an extremely randomized tree (ERT) classifier Geurts et al. (2006) by
fitting several decision-tree classifiers on a random subset of features with
a randomly defined threshold to prevent overfitting. While RF and ERT are
meta-algorithms based on the bagging technique, XGBoost Chen and Guestrin
(2016) follows boosting approach to ensemble construction and focuses on
incorrect predictions. It varies from Random Forest such that XGBoost always
prioritizes functional space while reducing the cost of a model, whereas
Random Forest tries to prioritize hyperparameters when optimizing the model.
McGuire et al. (2019) uses XGBoost for window-based feature extraction from
time series of physical parameters to classify solar flares. However the
aforementioned ensembles can optimize on one set of data modality.

Besides decision trees, different models trained with different algorithms but
with same data modalities can also be used in an ensemble as in Liu et al.
(2017b). However, they only included magnetograms with ARs within $\pm$30∘ of
the central meridian of the Sun for $\geq$C1.0-class flares and then designed
a multimodel integrated learner (MIM) by fitting several distinct base
learners, such as neural networks, naive classifiers, and SVMs. Finally, the
outputs of base learners were combined by a genetic algorithm. Similar efforts
for $\geq$C1.0-class flares forecasting can be seen in Campi et al. (2019)
where ARs extracted from SDO/HMI images from 2012 September 14 and 2016 April
30 are used and two-third of the instances are randomly selected for training
and one-third for testing their models. Furthermore, in Domijan et al. (2019)
they study the predictive capabilities of magnetic-feature properties located
within $\pm$45∘ from the solar central meridian and detected using Solar
Monitor Active Region Tracker Higgins et al. (2011) in Michelson Doppler
Imager (MDI) magnetograms and analyze the features to predict $\geq$C1.0-class
flares within the 24 hours following the observation. In this data-driven era
of predictive models, complex models can bring on higher accuracy, but also
ensembles allow many weak models to be combined to produce a meta model that
can compete with the state-of-the-art research efforts Murray (2018).

In recent years, the usage of ensembles have become a more popular research
topic in space weather forecasting. Guerra et al. (2015) created a multi-model
ensemble from four base learners for $\geq$M1.0-class flare prediction,
finding an improved forecast output compared to any one single model.
Similarly, Schunk et al. (2016) built an ionosphere-thermosphere-
electrodynamics multimodel ensemble prediction system based on seven physics-
based data assimilation models. Furthermore, in Guerra et al. (2020), full-
disk probabilistic forecasts from six operational forecasting methods are
converted to an ensemble for $\geq$M1.0-class flares by a linear classifier
and create a total of 28 ensembles to show the improvement of such a technique
over individual model forecasts. Although, ensemble methods are increasingly
being used by space weather researchers, much of this research has yet to be
implemented into operations, where transitioning comes with issues of model
compatibility.

It is worth noting that using a flare forecasting model in operational
settings, generally it is preferred to use more simplistic robust methods.
Diving into meteorology’s scenario, The NASA Community Coordinated Modeling
Center’s (CCMC) CME Scoreboard
(111https://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard/) and solar flare
Scoreboard (222https://ccmc.gsfc.nasa.gov/challenges/flare.php) provide an
weighted and equi-weight average of multiple forecast scores. Using an equal
weighted average of multiple forecasts can be used as a reliable first guess
over a more complex model runs or deciding on one specific forecast out of
several in operations Murray (2018), however, an ensemble derived from a
linear combination of multiple models can add to the decision making
capabilities on one final forecast leveraging the advantage of simplicity and
hence making it more reliable to trust its decision while in operation.

To evaluate a flare forecasting system in an operational scenario, Cinto et
al. (2020) provides a set of criteria that are worth considering and can be
used to distinguish a non-operationally evaluated system: (i) model evaluation
without truly unseen data, (ii) using active region (AR) magnetograms only
near the center of the solar disk, (iii) only using AR magnetograms linked to
$\geq$C1.0-class flares, and (iv) using insufficient data instances. The
author argues that the non-operationally evaluated system are evaluated under
certain bias and that does not make them wrong, however, evaluating under such
specific conditions might impair their predictive capabilities in real
operational settings. In addition to these guidelines, it is essential to note
that, most of the studies, create a cross-validation dataset by randomizing
the process of data splitting. While such data splitting leads to higher
experimental accuracy scores, it often fails to deliver similarly real-time
performance as discussed in Ahmadzadeh et al. (2021). We build our models that
meet the standard of the aforementioned criteria as they can address the near-
limb events with the full-disk base learner, they are trained and tested with
a time-segmented partitioning of data from solar cycle 24, and we evaluate our
models using data instances that were not presented to the models during
training to address the operational settings of flare forecasting.

In this work, we combine the prediction probabilities of two types of base
learners by the means of a linear classifier based on logistic regression. Our
first base learner, which is a deep learning based model which focuses on
spatial variation of a full-disk magnetogram. Similarly, our second base
learner is a heuristic-based aggregation model which outputs full disk
probability using the results from active region-based multivariate time
series classifiers. We train and validate an operations-ready ensemble flare
prediction model which optimizes the predictive performance of both our base
learners and provides a better confidence while issuing a flare forecast.

##  4 Methodology

Ensemble approaches integrate multiple forecasts into a single prediction by
combining the predictions from multiple base learners. A simplistic way of
integrating the forecasts is to use an equal weighting for each forecast and
combine to improve on a single-model prediction which we use as our baseline
meta-model. As mentioned earlier, we attempt to combine the predictions of two
base learners: (i) a deep learning-based full-disk flare predictor using
Helioseismic and Magnetic Imager (HMI) instrument onboard Solar Dynamics
Observatory (SDO) images and (ii) a multivariate time series classifier
utilizing magnetic field metadata to issue one combined full-disk flare
forecast.

###  4.1 Base Learners

####  4.1.1 Time-Series Forest

Our active region-based prediction model is a multivariate Time Series Forest
(TSF), trained with Space Weather Analytics benchmark dataset for solar flare
prediction (SWAN-SF) Angryk et al. (2020a, b) to predict the occurrence of
$\geq$M1.0-class flares within the next 24 hours by using an observation
window of 12 hours. The SWAN-SF is an open source multivariate time series
(MVTS) dataset that provides time series instances for a collection of space
weather related physical parameters within $\pm$70∘ primarily calculated for
each active regions from solar photospheric magnetograms. The TSF model is
trained by utilizing six magnetic-field parameters: (i) TOTUSJH (Total
unsigned current helicity), (ii) TOTPOT (Total photospheric magnetic free
energy density), (iii) TOTUSJZ (Total unsigned vertical current), (iv) ABSNJZH
(Absolute value of the net current helicity), (v) SAVNCPP (Sum of the modulus
of the net current per polarity), and (vi) USFLUX (Total unsigned flux) from
the suggested list of 13 parameters in Bobra and Couvidat (2015) as these are
available in near-real time, which is a necessity for an operational system.
The model outputs the flaring probability for an individual active region and
the implementation of this model is based on Ji et al. (2020).

####  4.1.2 Deep Learning Model

We trained an AlexNet-based Krizhevsky et al. (2012) Convolutional Neural
Network to perform full-disk binary flare prediction for $\geq$M1.0-class
flares. Similar to the active region-based counterparts, the full-disk model
assumes a 24 hour prediction window, but uses a single image (point-in-time
observation) to perform the predictions. For this task, we collected
compressed 8-bit images created from full-disk line-of-sight magnetograms
provided by HMI/SDO. We collected two compressed magnetogram images per day
(bi-daily image samples) at 00:00 UT and 12:00 UT from December 2010 to
December 2018 using Helioviewer API Muller et al. (2009) and labeled them
based on maximum of GOES peak X-ray flux converted to NOAA/GOES flare classes
observed in next 24 hours as shown in Fig.1. Unlike the TSF model, this deep
learning model outputs flaring probability for the entire full-disk and its
implementation is based on Pandey et al. (2021).

Figure 1: A timeline diagram to present the problem formulation of our deep
learning-based full-disk flare prediction model using bi-daily observations of
full-disk line-of-sight magnetograms and prediction window of 24 hours
considered to label the magnetogram instances.

We used trimonthly partitioning for training our models, which is non-
chronological time-segmented partitioning strategy, where Partition-1 contains
data from January to March, Partition-2 from April to June, Partition-3 from
July to September, and Partition-4 from October to December in a timeline from
2010 to 2018. The AR-based model also uses the same partitioning for aligning
our training partitions and avoiding the penetration of training partitions
into testing data in different prediction modalities to ensure the fair
comparisons and avoid partial memorization through temporal coherence
Ahmadzadeh et al. (2021).

###  4.2 Flare Prediction Ensemble

Our active region-based model outputs probabilities of flare ($P_{FL}$) for
each active region which we then aggregate to obtain a restricted full-disk
flaring probability (i.e., from active regions in central locations). We use
the following heuristic function in Eq. (1) to determine aggregated active
region flaring probability.

$P_{aggregated}=1-\prod_{i}\big{[}1-{P_{FL}(AR_{i})\big{]}}$ (1)

, where PFL (ARi) is the flaring probability of an active region, and the
aggregated result calculates the probability of having at least one flaring
active region, assuming the flaring events from active regions are
conditionally independent. The product term calculates the probability of
having no flaring active regions. These aggregated results from the active-
region based model are then concatenated with full-disk model’s output. The
aggregation procedure searches for most-recent valid active-region predictions
up to 6 hours prior to the designated forecast issue time. These gathered
predictions from full-disk and aggregated full-disk probabilities are then
combined to issue a final flare forecast using an ensemble. In this work,
while preparing our final dataset for the full-disk model, we do not include
magnetogram images where the observation time of the available image and
requested image timestamp is more than six hours. Therefore due to data
unavailability through helioviewer, we have used a total of 4,235 data
instances, where 3,502 are No Flare (NF) instances and 733 are Flare (FL)
instances. The detailed distribution of the dataset for each tri-monthly
partition is shown in Fig. 4 and the class imbalance ratios across the
partitions are generally consistent from $\sim$ 12 to 22% ($\sim$3.6:1 to
$\sim$7.2:1).

Figure 2:  Total number of Flare and No Flare instances across 4 trimonthly
partitions used in this work444We note that, while aggregating active regions
based outputs to full-disk probabilities, there were instances that were not
available even when we search for most-recent valid active-region predictions
up to 6 hours prior to the designated forecast. Therefore, such instances are
also removed from full-disk models for consistency.

In our baseline meta-model approach, we use equal weighted averaging of flare
probabilities from aggregated active-regions and full-disk flaring
probabilities for issuing a final forecast. In other words, given two flaring
probabilities from two approaches, the baseline approach is to compute the
arithmetic average of the probabilities, assuming equal importance. This
simplistic combination of flare probabilities will serve as our baseline,
although it is a naive approach that does not consider the intrinsic
characteristics of long-term diagnostic results from the models.

Our alternative approach to the baseline meta-model is logistic regression-
based classifier that is trained with flaring probabilities from the base
learners. As we already use two powerful algorithms to train our base learner
to extract the complex dynamics of the datasets, we chose a linear model,
logistic regression, because of its simplicity and computational efficiency
for the final prediction result.The infrastructure of our complete flare
prediction system design is presented in Fig. 3 which shows our overall
methodology for creating an ensemble using two heterogeneous base learners
that outputs a full-disk flare forecast.

Figure 3: An illustration of our ensemble flare prediction pipeline showing
two base learners (AR-based FP) and (Full-disk FP) and the ensemble (Meta-FP)
followed by full-disk aggregation of AR-based FP’s flare probabilities

Given the flare probability scores of two base learners which we utilize as
two input features — PFL(FD) and PFL(Aggregated), and one binary (0/1) target
feature (y) where 0 is used for No flare (NF) and 1 is used for Flare (FL).
Logistic Regression aim to optimize the weights ($w_{1}$, $w_{2}$, and $b$),
such that:

$Z=w_{1}\times P_{FL}(FD)+w_{2}\times P_{FL}(Aggregated)+b$ (2)
$\hat{y}=\sigma(Z)$ (3)

where, $Z$ in Eq. ( 2) is the linear combination of two base learners’ output,
$\sigma$ is the sigmoid activation function, and $\hat{y}$ is the predicted
output as shown in Eq. ( 3). The above problem of finding the optimized
weights $w_{1}$, $w_{2}$ for two base learners is formulated as an
optimization problem where the loss is minimized to get the better values of
weights using a logistic loss function as shown in Eq. ( 4).

$loss(L)=-\frac{1}{N}\sum_{i=1}^{N}\big{[}(y_{i}\cdot
log(\hat{y}_{i}))+((1-y_{i})\cdot log(1-\hat{y}_{i}))\big{]}$ (4)

We use stochastic gradient descent (SGD) as our solver for the optimization
with hyperparameter tuning. The hyperparameters we considered are learning
rate and different regularization parameters which includes L1 loss Tibshirani
(1996), L2 loss Hoerl and Kennard (1970), and linear mixings of L1 and L2 loss
Zou and Hastie (2005). and As we will describe later on Sec.5, we employ
2-fold cross-validation for our meta-model where we use one of the test
partition scores of the base learners to train and another for testing our
meta model, referred to as Meta-FP, interchangeably. We note that we aim to
provide full-disk forecasts by computing the aggregated flare probability
scores from active regions to make it compatible with the full-disk model
using the probabilistic heuristic shown in Eq. ( 1).

##  5 Experimental Evaluation

###  5.1 Experimental Settings

In this work, we trained two base learners for flare prediction
($\geq$M1.0-class flares) with two different dataset and model configurations
and architectures. Although our two base learners utilize two different data
modalities (i.e., point-in-time image and multivariate time series), we used
time-segmented tri-monthly partitioning when training both of these models. We
divided our datasets into four partitions to ready our 3-fold holdout cross-
validation dataset. The data in Partition-1 contains images from the months of
January to March, Partition-2 from April to June, Partition-3 from July to
September, and Partition-4 from October to December. Here, this partitioning
of the dataset is created by dividing the data timeline from Dec 2010 to Dec
2018 into four partitions on the basis of months rather than chronological
partitioning, to incorporate approximately equal distribution of flaring
instances in every fold for training , validating, and testing the model.
Furthermore, such a partitioning strategy diversify the data instances in both
the training and testing phase of our models as it considers instances during
solar maxima and minima of solar cycle 24 used in this work.

We create three sets of base learner models from 3-fold cross-validation
experiments as our base learners where we use Partition-3 as our hold-out test
set (i.e., never used in training and validation). Then,

  * •

In Fold-1, we trained both of our base learners with Partition-1 and
Partition-2 and validated on Partition-4

  * •

In Fold-2, we trained both of our base learners with Partition-1 and
Partition-4 and validated on Partition-2

  * •

In Fold-3, we trained both of our base learners with Partition-2 and
Partition-4 and validated on Partition-1.

All of these three base learners are tested on Partition-3. Partition-3 as a
test differs from the validation sets in each fold such that, we used the
validation set in every epoch to track the performance of our model whereas
the test set, Partition-3, is used only once to confirm the performance of the
trained models and meta-models at the end.

To train and validate our Meta-FP, we create our dataset based on the
probability scores of our three base learner sets obtained from 3-Fold cross
validation experiments. The details of our experimental design is shown in
Fig. 4. We used the flare probability scores from the validation set and test
set used in respective base learners interchangeably to train and validate our
Meta-FP model which is a general linear model i.e., logistic regression (LR).
The experiments for Meta-FP are performed in such way that:

  * •

In Expt. 1, we performed 2-fold cross validation with Partition-4 and
Partition-3.

  * •

In Expt. 2, we performed 2-fold cross validation with Partition-2 and
Partition-3.

  * •

In Expt. 3, we performed 2-fold cross validation with Partition-1 and
Partition-3.

In doing so, we trained six Meta-FP models based on logistic regression and
compared our results with a baseline Meta-FP which is an equal weighted
average of two base learners.

Figure 4: An experimental design diagram to depict the flow of our experiments
for this work. Meta-FP experiments for logistic regression (LR) are cross-
validated using each fold results of base learners. This results into 2-fold
cross-validation in each experiments of Meta-FP.

To evaluate the performance of our models, we create a contingency matrix,
which includes information on True Positives (TP), True Negatives (TN), False
Positives (FP) and False Negatives (FN) to evaluate the performance of our
base learners and Meta-FP. Note that, in the context of our flare prediction
task, Flare (FL) is considered as the positive outcome while No Flare (NF) is
the negative. Using these four outcomes we use two widely used performance
metrics in space weather forecasting, True Skill Statistics (TSS, shown in Eq.
( 5)) and Heidke Skill Score (HSS, shown in Eq. ( 6)) to evaluate our model.

$TSS=\frac{TP}{TP+FN}-\frac{FP}{FP+TN}$ (5)

$HSS=2\times\frac{TP\times TN-FN\times FP}{((P\times(FN+TN)+(TP+FP)\times
N))}$ (6)

The values of TSS range from -1 to 1, where 1 indicates all correct
predictions, -1 represents all incorrect predictions, and 0 represents no-
skill, often transpiring as the random or one-sided (all positive/all
negative) predictions. It is defined as the difference between True Positive
Rate (TPR) and False Positive Rate (FPR) and does not account for class-
imbalance, i.e., treats false positives (FP) and false negatives (FN) equally.
Similarly, HSS measures the forecast skill of the models over an imbalance-
aware random prediction. It ranges from -$\infty$ to 1, where 1 represents the
perfect skill and 0 represents no skill gain over a random prediction. It is
common practice to use HSS for the solar flare prediction models (similar to
weather predictions where forecast skill has more value than accuracy or
single-class precision), due to the high class-imbalance ratio present in the
datasets.

Figure 5: Validation Scores of base learners in Fold-1 (base learners trained
with Partitions 1 and 2) and the corresponding validation scores of Meta-FP
(meta models trained in Expt. 1).

###  5.2 Evaluation

Although AR-based classifiers are better for pinpointing the source active
regions for flares and giving more accurate estimations for forecasting
flaring phenomena, the aggregated results drop significantly in contrast to
our expectation. The results from AR-based models shows TSS=0.82$\pm$0.02 and
HSS=0.20$\pm$0.04 when these methods are evaluated solely on active region
based confusion matrices. However, when we aggregate them, these models fail
to reach the acceptable levels of skill scores as they drop to
TSS=0.32$\pm$0.04 and HSS=0.15$\pm$0.02. The reason for these issues may stem
from three reasons: (i) limb events are not considered (beyond $\pm$70∘) as
there are no reliable magnetic field readings, (ii) these models are not
optimized for full-disk flare prediction, and/or (iii) an independent, equally
weighted aggregation scenario in our heuristic approach. Furthermore, the drop
in aggregated skill scores can be attributed to the number of high false
positives, which is common in rare-event forecasting problems and particularly
in flare forecasting. The reason we empirically observed throughout the years
for these false positives are often the models’ inability to distinguish [C4+
to C9.9] flares from $\geq$M-class flares as discussed in Pandey et al.
(2022). All in all, our first observation is that for full-disk flare
prediction, our designated deep learning models are more effective when
compared to the AR aggregations as it considers the near-limb events by using
a compressed full-disk magnetogram which are suitable to capture the shape
parameters in the active regions within and beyond $\pm$70∘ of the central
meridian of the Sun.

Figure 6: Validation Scores of base learners in Fold-2 (base learners trained
with Partitions 1 and 4) and the corresponding validation scores of Meta-FP
(meta models trained in Expt. 2). Figure 7: Validation Scores of base learners
in Fold-3 (base learners trained with Partitions 2 and 4) and the
corresponding validation scores of Meta-FP (meta models trained in Expt. 3).

Analyzing our results, we observed that our logistic regression-based Meta-FP
improves on both TSS and HSS compared to two base learners and equal weighting
baseline meta learner on respective test partitions as shown in Fig. 5, 6, 7.
In our first experiment, we trained two Meta-FP models that utilizes the flare
probability scores of two base learners that are trained with Partition-1 and
Partition-2 of the respective datasets. We train and validate our Meta-FP with
respect to the unused two partitions that are Partition-3 and Partition-4 for
the first experiment as shown in Fig. 5. Our other two experiments are also
consistent with making sure to only use two such partitions that have not been
used while training the base learners as shown in Fig. 6 and 7. While the
improvement in terms of TSS and HSS on both the base learner and baseline
Meta-FP can be seen across all six logistic regression-based Meta-FP model,
the maximum improvement of logistic regression over base learners and baseline
can be seen with base learners in Fold-1 (trained with Partition-1 and
Partition-2) where the Meta-FP is trained with Partition 3 and tested on
Partition-4 (right side of the Fig. 5). In this experiment, the logistic
regression model improves on full-disk (base learner) in terms of TSS by
$\sim$6% and HSS by $\sim$14%. Similarly, it improves on aggregated AR-based
models in terms of TSS by $\sim$22% and HSS by $\sim$28%. While we used the
equal weighted averaging as a baseline model, it does not improve on the
results from the full-disk base learner. However, compared to the baseline for
the same experiment (Fold-1) as explained above, the logistic regression model
improves by $\sim$13% and $\sim$21% in terms of TSS and HSS respectively.

On an average, we observe that our full-disk model (base learner) has
TSS=0.40$\pm$0.07 and HSS=0.25$\pm$0.07 and the AR-based model (base learner)
has TSS=0.32$\pm$0.04 and HSS=0.15$\pm$0.02 computed over both test and
validation results from all three folds. When we employed the baseline meta
learner (equal-weighted average), the average TSS=0.39$\pm$0.05 and
HSS=0.20$\pm$0.04 is observed. Given that, equal weighted average is used as a
common way to ensemble two or more models, it can be problematic as it could
not even surpass the scores of a base learner (full-disk model). With the
logistic regression-based meta learner (Meta-FP), the average TSS and HSS
observed is 0.49$\pm$0.02 and 0.35$\pm$0.05 respectively. Therefore, we see
that on an average, the Meta-FP improves on the full-disk model by $\sim$9% in
terms of TSS and $\sim$10% in terms of HSS. Similarly, it improves on the AR-
based model by $\sim$17% and $\sim$20% in terms of TSS and HSS respectively.
Finally, when compared to the baseline meta model, it improves on TSS by
$\sim$10% and HSS by $\sim$15%.

##  6 Discussion

Ensemble methods combines multiple models to obtain better predictive
performance than could be obtained from any of the constituent model alone. By
using an ensemble method, we learn how the single model output can be improved
based on (a) maximum voting, (b) equal weighted averaging, and (c) weighted
voting. Learning the weights in weighted voting, in the scope of this paper,
is structured as a logistic regression problem. One usual way to create an
ensemble is to simply average the forecast probabilities of multiple models
and provide a final forecast decision, however, it is naive to assume that all
base-learners are equally good. Therefore, the main objective of training an
ensemble here is to learn and assign better weights for two base-learner
predictions by quantifying the level of impact of individual models
predictions on the final forecast. The prediction distribution for Partition-3
and Partition-4 used in Experiment-1 for training and testing the ensemble
alternatingly and the learned decision-boundary by Meta-FP LR is shown in Fig.
8 as an example to show how an ensemble improves over the base-learner by
coupling using a linear classifier. The predicted probability distribution and
learned decision boundary in Experiment-2 and 3 is presented in Fig. S1 and S2
respectively. Furthermore, the confusion matrices for base-learners
predictions and for the consequent ensembles created in all three experiments
are presented in Table S1, S2, S3, S4, S5, and S6.

Figure 8: The figure shows the distribution of predicted probability scores
for all the data instances used in Experiment-1 and the decision boundary
learned by the trained logistic regression classifier for both the partitions:
(a) Distribution of probabilities scores in partition-3 of two base learner
mapping to actual ouput. (b) Distribution of probabilities scores in
partition-4 of two base learner mapping to actual ouput. (c) The learned-
decision boundary by Meta-FP LR while training on partition-4. (d) The
learned-decision boundary by Meta-FP LR while training on partition-3. (e) The
learned-decision boundary by Meta-FP LR validated on partition-3. (f) The
learned-decision boundary by Meta-FP LR validated on partition-4.

Ensemble methods defy the idea of making one model and relying on this model
as the best/most accurate predictor we can make. It rather take a multitude of
models into account, and combine those models to produce one final model that
issues a final forecast. At this point, we do have access to very complex
machine learning paradigms that have proven to be very effective in several
areas, such as computer vision and image classification. However, relying on
the forecast of a single model for rare events like major solar flares might
be critical for a system in operation. The model thus obtained might be biased
on the dataset used to train the model and can be just as good as the curated
dataset used to create the model. Therefore, it is essential to have a
reliable flare forecasting model obtained by assembling multiple models with
different data modalities to leverage the most with coupling.

##  7 Conclusion and Future Work

In this work, we trained a logistic regression-based meta learner for flare
prediction that combines the probabilities of two flare prediction models
trained with different datasets and machine learning paradigms. While we have
two models (base learners) with their own advantages in prediction
capabilities, we observed that for base learners, full disk models have better
performance for full disk flare forecasting compared to AR-aggregation.
Therefore, with a motive of further improving the performance of base
learners, we explored a simplest way to combine them by training an ensemble
flare predictor which automates the task of assigning weights to the outputs
of our base learners, thus improving the overall performance of our models and
adding robustness to the prediction task compared to equal weighted
ensembling.

Furthermore, considering that we only used bi-daily observations, the shape
parameters considered in compressed magnetograms proves to be actually
powerful. AR-based models on the other hand, using magnetic field data, either
as images or derived products, as they are now, will have limited capability
although they have higher sensitivity per active region. Therefore, a
complementary approach is necessary that does not only rely directly on
magnetic field rasters and this work introduces a technique which considers
both the magnetic-field parameters and shape-based parameters to obtain flare
forecasting models with their own essence and abilities. Finally, we combine
these two heterogeneous models into one coupled model using a linear ensemble
to improve overall performance. Although we see significant improvements in
skill scores after ensembling, our coupled models are not without limitations
that are also inherited from our full-disk based model trained with point-in-
time bi-daily observations, which overlooks the temporal evolution of
magnetic-field parameters of the active regions which can limit the predictive
capabilities of full-disk flare predictors. Therefore, our next goal is to
formulate the flare prediction task as a video classification problem using
full-cadence image sequences that will account for the temporal evolution of
active regions. Furthermore, there are several other directions that can be
explored such as using a basis function on the aggregated active region
prediction probabilities, finding other better aggregation strategies that
could boost the performance of AR-based models while computing a full-disk
probability and elaborate the ensemble using more sophisticated classifiers,
aiming to further improve the predictive capabilities of our models.

## Funding

This project is supported in part under two NSF awards #2104004 and #1931555
jointly by the Office of Advanced Cyberinfrastructure within the Directorate
for Computer and Information Science and Engineering, the Division of
Astronomical Sciences within the Directorate for Mathematical and Physical
Sciences, and the Solar Terrestrial Physics Program and the Division of
Integrative and Collaborative Education and Research within the Directorate
for Geosciences.

## Acknowledgments

The data used in this project is a courtesy of NASA/SDO and the AIA, EVE, and
HMI science teams 555https://sdo.gsfc.nasa.gov/data/. We also want to thank
the developers of Helioviewer Project 666https://api.helioviewer.org/docs/v2/
for providing an API for input images.

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.

## Author Contributions

CP built deep learning-based full-disk models, ensemble models, contributed in
design of experiments and writing the manuscript. AJ built AR-based
classifiers and contributed on reviewing the manuscript. RA was involved in
planning and participated in writing the manuscript. MG provided the domain
expertise, contributed in model coupling concepts, and reviewing the
manuscript. BA conceived of the original idea, and contributed in design of
experiments, model coupling concepts, planning, and writing the manuscript.

## Data Availability Statement

The source code and datasets analyzed for this study can be found in the
following bitbucket repository:
https://bitbucket.org/gsudmlab/ensemble_fp/src/main/revised_expt/

## References

  * Abduallah et al. (2021) Abduallah, Y., Wang, J. T. L., Nie, Y., Liu, C., and Wang, H. (2021).  DeepSun: machine-learning-as-a-service for solar flare prediction.  _Research in Astronomy and Astrophysics_ 21, 160.  10.1088/1674-4527/21/7/160
  * Ahmadzadeh et al. (2021) Ahmadzadeh, A., Aydin, B., Georgoulis, M. K., Kempton, D. J., Mahajan, S. S., and Angryk, R. A. (2021).  How to train your flare prediction model: Revisiting robust sampling of rare events.  _The Astrophysical Journal Supplement Series_ 254, 23.  10.3847/1538-4365/abec88
  * Angryk et al. (2020a) [Dataset] Angryk, R., Martens, P., Aydin, B., Kempton, D., Mahajan, S., Basodi, S., et al. (2020a).  SWAN-SF.  10.7910/DVN/EBCFKM
  * Angryk et al. (2020b) Angryk, R. A., Martens, P. C., Aydin, B., Kempton, D., Mahajan, S. S., Basodi, S., et al. (2020b).  Multivariate time series dataset for space weather data analytics.  _Scientific Data_ 7\.  10.1038/s41597-020-0548-x
  * Aso et al. (1994) Aso, T., Ogawa, T., and Abe, M. (1994).  Application of back-propagation neural computing for the short-term prediction of solar flares.  _Journal of geomagnetism and geoelectricity_ 46, 663–668.  10.5636/jgg.46.663
  * Benvenuto et al. (2018) Benvenuto, F., Piana, M., Campi, C., and Massone, A. M. (2018).  A hybrid supervised/unsupervised machine learning approach to solar flare prediction.  _The Astrophysical Journal_ 853, 90.  10.3847/1538-4357/aaa23c
  * Bobra and Couvidat (2015) Bobra, M. G. and Couvidat, S. (2015).  Solar flare prediction using sdo/hmi vector magnetic field data with a machine-learning algorithm.  _The Astrophysical Journal_ 798, 135.  10.1088/0004-637x/798/2/135
  * Breiman (1996) Breiman, L. (1996).  Bagging predictors.  _Machine Learning_ 24, 123–140.  10.1023/a:1018054314350
  * Breiman (2001) Breiman, L. (2001).  _Machine Learning_ 45, 5–32.  10.1023/a:1010933404324
  * Campi et al. (2019) Campi, C., Benvenuto, F., Massone, A. M., Bloomfield, D. S., Georgoulis, M. K., and Piana, M. (2019).  Feature ranking of active region source properties in solar flare forecasting and the uncompromised stochasticity of flare occurrence.  _The Astrophysical Journal_ 883, 150.  10.3847/1538-4357/ab3c26
  * Chen and Guestrin (2016) Chen, T. and Guestrin, C. (2016).  XGBoost.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (ACM).  10.1145/2939672.2939785
  * Cinto et al. (2020) Cinto, T., Gradvohl, A. L. S., Coelho, G. P., and da Silva, A. E. A. (2020).  A framework for designing and evaluating solar flare forecasting systems.  _Monthly Notices of the Royal Astronomical Society_ 495, 3332–3349.  10.1093/mnras/staa1257
  * Domijan et al. (2019) Domijan, K., Bloomfield, D. S., and Pitié, F. (2019).  Solar flare forecasting from magnetic feature properties generated by the solar monitor active region tracker.  _Solar Physics_ 294\.  10.1007/s11207-018-1392-4
  * Feng et al. (2020) Feng, L., Gan, W., Liu, S., Wang, H., Li, H., Xu, L., et al. (2020).  Space weather related to solar eruptions with the aso-s mission.  _Frontiers in Physics_ 8\.  10.3389/fphy.2020.00045
  * Florios et al. (2018) Florios, K., Kontogiannis, I., Park, S.-H., Guerra, J. A., Benvenuto, F., Bloomfield, D. S., et al. (2018).  Forecasting solar flares using magnetogram-based predictors and machine learning.  _Solar Physics_ 293\.  10.1007/s11207-018-1250-4
  * Freund and Schapire (1996) Freund, Y. and Schapire, R. E. (1996).  Experiments with a new boosting algorithm.  In _IN PROCEEDINGS OF THE THIRTEENTH INTERNATIONAL CONFERENCE ON MACHINE LEARNING_ (Morgan Kaufmann), 148–156
  * Georgoulis et al. (2019) Georgoulis, M. K., Nindos, A., and Zhang, H. (2019).  The source and engine of coronal mass ejections.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 377, 20180094.  10.1098/rsta.2018.0094
  * Geurts et al. (2006) Geurts, P., Ernst, D., and Wehenkel, L. (2006).  Extremely randomized trees.  _Machine Learning_ 63, 3–42.  10.1007/s10994-006-6226-1
  * Guerra et al. (2020) Guerra, J. A., Murray, S. A., Bloomfield, D. S., and Gallagher, P. T. (2020).  Ensemble forecasting of major solar flares: methods for combining models.  _Journal of Space Weather and Space Climate_ 10, 38.  10.1051/swsc/2020042
  * Guerra et al. (2015) Guerra, J. A., Pulkkinen, A., and Uritsky, V. M. (2015).  Ensemble forecasting of major solar flares: First results.  _Space Weather_ 13, 626–642.  10.1002/2015sw001195
  * Higgins et al. (2011) Higgins, P., Gallagher, P., McAteer, R., and Bloomfield, D. (2011).  Solar magnetic feature detection and tracking for space weather monitoring.  _Advances in Space Research_ 47, 2105–2117.  10.1016/j.asr.2010.06.024
  * Hoeksema et al. (2014) Hoeksema, J. T., Liu, Y., Hayashi, K., Sun, X., Schou, J., Couvidat, S., et al. (2014).  The helioseismic and magnetic imager (HMI) vector magnetic field pipeline: Overview and performance.  _Solar Physics_ 289, 3483–3530.  10.1007/s11207-014-0516-8
  * Hoerl and Kennard (1970) Hoerl, A. E. and Kennard, R. W. (1970).  Ridge regression: Biased estimation for nonorthogonal problems.  _Technometrics_ 12, 55–67.  10.1080/00401706.1970.10488634
  * Huang et al. (2018) Huang, X., Wang, H., Xu, L., Liu, J., Li, R., and Dai, X. (2018).  Deep learning based solar flare forecasting model. i. results for line-of-sight magnetograms.  _The Astrophysical Journal_ 856, 7.  10.3847/1538-4357/aaae00
  * Ji et al. (2020) Ji, A., Aydin, B., Georgoulis, M. K., and Angryk, R. (2020).  All-clear flare prediction using interval-based time series classifiers.  In _2020 IEEE International Conference on Big Data (Big Data)_ (IEEE), 4218–4225.  10.1109/bigdata50022.2020.9377906
  * Jonas et al. (2018) Jonas, E., Bobra, M., Shankar, V., Hoeksema, J. T., and Recht, B. (2018).  Flare prediction using photospheric and coronal image data.  _Solar Physics_ 293\.  10.1007/s11207-018-1258-9
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. (2012).  Imagenet classification with deep convolutional neural networks.  _Neural Information Processing Systems_ 25\.  10.1145/3065386
  * Li et al. (2020) Li, X., Zheng, Y., Wang, X., and Wang, L. (2020).  Predicting solar flares using a novel deep convolutional neural network.  _The Astrophysical Journal_ 891, 10.  10.3847/1538-4357/ab6d04
  * Liu et al. (2017a) Liu, C., Deng, N., Wang, J. T. L., and Wang, H. (2017a).  Predicting solar flares using sdo/hmi vector magnetic data products and the random forest algorithm.  _The Astrophysical Journal_ 843, 104.  10.3847/1538-4357/aa789b
  * Liu et al. (2017b) Liu, J.-F., Li, F., Wan, J., and Yu, D.-R. (2017b).  Short-term solar flare prediction using multi-model integration method.  _Research in Astronomy and Astrophysics_ 17, 034.  10.1088/1674-4527/17/4/34
  * McGuire et al. (2019) McGuire, D., Sauteraud, R., and Midya, V. (2019).  Window-based feature extraction method using XGBoost for time series classification of solar flares.  In _2019 IEEE International Conference on Big Data (Big Data)_ (IEEE).  10.1109/bigdata47090.2019.9006212
  * Muller et al. (2009) Muller, D., Fleck, B., Dimitoglou, G., Caplins, B., Amadigwe, D., Ortiz, J., et al. (2009).  JHelioviewer: Visualizing large sets of solar images using JPEG 2000\.  _Computing in Science & Engineering_ 11, 38–47.  10.1109/mcse.2009.142
  * Murray (2018) Murray, S. A. (2018).  The importance of ensemble techniques for operational space weather forecasting.  _Space Weather_ 16, 777–783.  10.1029/2018sw001861
  * Nishizuka et al. (2021) Nishizuka, N., Kubo, Y., Sugiura, K., Den, M., and Ishii, M. (2021).  Operational solar flare prediction model using deep flare net.  _Earth, Planets and Space_ 73\.  10.1186/s40623-021-01381-9
  * Nishizuka et al. (2018) Nishizuka, N., Sugiura, K., Kubo, Y., Den, M., and Ishii, M. (2018).  Deep flare net (DeFN) model for solar flare prediction.  _The Astrophysical Journal_ 858, 113.  10.3847/1538-4357/aab9a7
  * Nishizuka et al. (2017) Nishizuka, N., Sugiura, K., Kubo, Y., Den, M., Watari, S., and Ishii, M. (2017).  Solar flare prediction model with three machine-learning algorithms using ultraviolet brightening and vector magnetograms.  _The Astrophysical Journal_ 835, 156.  10.3847/1538-4357/835/2/156
  * Núñez and Paul-Pena (2020) Núñez, M. and Paul-Pena, D. (2020).  Predicting predicting ¿10 mev sep events from solar flare and radio burst data.  _Universe_ 6, 161.  10.3390/universe6100161
  * Pandey et al. (2021) Pandey, C., Angryk, R. A., and Aydin, B. (2021).  Solar flare forecasting with deep neural networks using compressed full-disk HMI magnetograms.  In _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE), 1725–1730.  10.1109/bigdata52589.2021.9671322
  * Pandey et al. (2022) Pandey, C., Angryk, R. A., and Aydin, B. (2022).  Deep neural networks based solar flare prediction using compressed full-disk line-of-sight magnetograms.  In _Information Management and Big Data_ (Springer International Publishing). 380–396.  10.1007/978-3-031-04447-2_26
  * Park et al. (2018) Park, E., Moon, Y.-J., Shin, S., Yi, K., Lim, D., Lee, H., et al. (2018).  Application of the deep convolutional neural network to the forecast of solar flare occurrence using full-disk solar magnetograms.  _The Astrophysical Journal_ 869, 91.  10.3847/1538-4357/aaed40
  * Schunk et al. (2016) Schunk, R. W., Scherliess, L., Eccles, V., Gardner, L. C., Sojka, J. J., Zhu, L., et al. (2016).  Space weather forecasting with a multimodel ensemble prediction system (MEPS).  _Radio Science_ 51, 1157–1165.  10.1002/2015rs005888
  * Tibshirani (1996) Tibshirani, R. (1996).  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ 58, 267–288.  10.1111/j.2517-6161.1996.tb02080.x
  * Toriumi and Wang (2019) Toriumi, S. and Wang, H. (2019).  Flare-productive active regions.  _Living Reviews in Solar Physics_ 16\.  10.1007/s41116-019-0019-7
  * Zou and Hastie (2005) Zou, H. and Hastie, T. (2005).  Regularization and variable selection via the elastic net.  _Journal of the Royal Statistical Society. Series B (Statistical Methodology)_ 67, 301–320