# Modeling and Exploration of Gain Competition Attacks in Optical Network-on-
Chip Architectures

Khushboo Rani  ID , Hansika Weerasena, Stephen A. Butler, Subodha Charles, and
Prabhat Mishra  ID
CISE, University of Florida, Gainesville, Florida, USA

###### Abstract

Network-on-Chip (NoC) enables energy-efficient communication between numerous
components in System-on-Chip architectures. The optical NoC is widely
considered a key technology to overcome the bandwidth and energy limitations
of traditional electrical on-chip interconnects. While optical NoC can offer
high performance, they come with inherent security vulnerabilities due to the
nature of optical interconnects.

In this paper, we investigate the gain competition attack in optical NoCs,
which can be initiated by an attacker injecting a high-power signal to the
optical waveguide, robbing the legitimate signals of amplification. To the
best of our knowledge, our proposed approach is the first attempt to
investigate gain competition attacks as a security threat in optical NoCs. We
model the attack and analyze its effects on optical NoC performance. We also
propose potential attack detection techniques and countermeasures to mitigate
the attack. Our experimental evaluation using different NoC topologies and
diverse traffic patterns demonstrates the effectiveness of our modeling and
exploration of gain competition attacks in optical NoC architectures.

###### Index Terms:

Network-on-Chip, Optical NoC, Gain Competition Attack

##  I Introduction

With the increasing levels of instruction, thread, and data-level parallelism
in CPUs, communication between diverse components in System-on-Chip (SoC)
architectures has become a major bottleneck from both performance and energy
perspectives. There has been a shift toward utilizing Network-on-Chip (NoC)
based communication in designing SoCs. Unlike bus-based architectures, NoC
offers a scalable and efficient solution [1]. In the early days, NoC design
relied on electrical interconnects to communicate between SoC components.
However, electrical interconnects face a serious challenge in keeping up with
the rate at which data must be transferred in the presence of increasing
parallelism and decreasing chip area. Drawing from the domain of
telecommunications, optical/photonic interconnects provide higher bandwidth,
better latency, reliability, and less energy usage compared to their
electrical counterpart [2]. Several prior studies explored optical
interconnects as a replacement for electrical on-chip interconnects [3] [2].
While Optical Network-on-Chip (ONoC) can offer high performance, it comes with
inherent security vulnerabilities due to the nature of optical interconnects.
SoC manufacturers use third-party Intellectual Property (IP) blocks in their
designs to reduce cost and meet time-to-market constraints. The long supply
chains coupled with the utilization of potentially untrusted IPs increases the
likelihood of different types of security threats, such as hardware Trojans,
for eavesdropping or degrading Quality-of-Service (QoS) [4] [5].

In the domain of optical communications, Erbium-Doped Fiber Amplifiers (EDFAs)
or Semiconductor Optical Amplifiers (SOA) [6] are used to periodically
strengthen an optical signal to counteract attenuation and signal drops [7].
Similarly, optical interconnects on the chip level can also experience
attenuation and amplitude drops. Grani and Bartolini [8] quantified these
losses that can happen over the optical waveguide and at different points in
the network. Such drops include a 0.02dB drop at a key component in the ONoC,
called the Microring Resonator (MR). MRs are critical to filtering wavelengths
in the photonic modulators, which are responsible for electrical to optical
conversions [2]. Amplitude drops can be more severe in Dense Wavelength-
Division Multiplexed (DWDM) networks when there are potentially $64$ or $128$
MRs (one for every wavelength) at one point in the network, causing a drop of
1.28 ($0.02\times 64$) or 2.56 ($0.02\times 128$) dB per node that the light
passes through. Other drops include a waveguide propagation loss of $1$ dB/cm
and a $0.7$ dB loss per waveguide cross [8]. These losses introduce the need
for optical amplifiers at the chip scale so that signals can remain at a
reasonable power level for photodetectors.

Figure 1: Gain Competition Attack: The malicious wavelength ($\lambda_{3}$)
gets higher amplification, which steal the power needed for amplification of
the legitimate wavelengths ($\lambda_{1},\lambda_{2}$, and $\lambda_{4}$).

###  I-A Threat Model

A gain competition attack occurs when a malicious party injects a powerful
optical signal into the network, and on amplification, the signal gets
intensified more than the legitimate signals. As a result, the legitimate
signals will be “drowned out” leading to increased packet retransmissions and
reduced Quality-of-Service (QoS). We examine this phenomenon in detail to
highlight the effect of the gain competition attack on the Signal-to-Noise
Ratio (OSNR) and Bit Error Rate (BER) in an ONoC architecture. OSNR is a key
parameter to estimate ONoC performance. It suggests a degree of impairment
when the optical signal is carried by an optical transmission system that
includes optical amplifiers. While optical amplifiers are essential to optical
networks, they also have inherent vulnerabilities. Amplifiers inherently
degrade the OSNR as signals pass through them. The manufacturer specifies the
amplifier’s noise figure, which quantifies how much the signal degrades after
amplification. This means that there will always be a lower OSNR after
amplification. The stronger signals receive more gain on amplification. This
leads to a vulnerability called gain competition, where stronger signals
receive more gain and can “drown out” the weaker signals [2].

Figure 1 shows how the OSNR is affected in a gain competition attack. Consider
initial OSNR ($OSNR_{0}$), which is an inherent property of the laser. At
point $X$, we see the malicious member of the network injecting some strong
signal on an out-of-band wavelength (represented by the red arrow). Note that
the malicious signal may not be in the same band as the legitimate signal. An
attacker can use a signal closer to a legitimate signal for maximum impact of
the attack [9]. At the amplifier $Y$, the gain is stolen by the strong
wavelength, and a relatively small gain is given to the legitimate signals
(yellow, green, and blue wavelengths). The OSNR ($OSNR_{1}$) severely degrades
compared to the original $OSNR_{0}$ after the amplification. This leads to an
increased BER between points $Y$ and $Z$ and a drop in QoS [10].

###  I-B Research Contributions

There are various research efforts that examine different types of attacks
that can occur on ONoCs [11] [7]. While the gain competition attack is well
understood in the computer networks domain, it has not been explored in the
context of ONoC design. This paper focuses on modeling the gain competition
attack, exploring its impact on ONoCs, and proposing potentially effective
mitigation techniques. Specifically, this paper makes the following
contributions.

  1. 1.

We model gain competition attacks in optical NoCs using a cycle-accurate NoC
simulator, Noxim [12]. We incorporate the bit error rate with the model to
show the impact on system throughput and explore QoS and Denial-of-Service
(DoS) attacks in ONoCs.

  2. 2.

We simulate the effect of the attacks on the performance of ONoCs and present
our results to demonstrate the effectiveness of our modeling and exploration
framework using a wide variety of traffic patterns and network topologies. We
also discuss a few feasible mitigation techniques for the attacks in ONoCs.

##  II Background and Related Work

###  II-A Optical NoC (ONoC)

Figure 2: A DWDM-based photonic waveguide with modulators and detectors Figure
3: An instance of HT infected GI in 4 $\times$ 4 hybrid-NoC.

Optical NoC provides the same set of benefits as electrical NoCs on large
multi-core systems. In ONoC, the interconnects on the chip use light to send
data instead of electrical signals, outperforming their electrical counterpart
with near speed-of-light latency, increased bandwidth, and lower power
dissipation [2] [5]. ONoCs are often combined with electrical interconnects to
form a hybrid topology (hybrid-NoC). This allows nearby cores to communicate
electrically (like local streets) while optical communication is used (like a
highway) for fast transmission over relatively long distances. These optical
links are connected with an off-chip laser source. One of the key components
of ONoC is Gateway Interface (GI). Each GI in an ONoC is able to send and
receive data in the optical domain on multiple utilized carrier wavelengths.
Therefore, each GI has two major components: the bank of MR modulators and a
bank of detector MRs (DB), which are responsible for sending and receiving
optical signals. In a bank, each MR operates on a specific carrier wavelength,
which allows the utilization of the excellent wavelength selectivity of MRs
and DWDM capability of waveguides to enable high bandwidth parallel data
transfers in ONoCs [13]. The signals are sent along the waveguide to the
destination GI, where the DB transforms the optical signals back to electrical
ones. Figure 2 shows an example of a GI, with 4 MRs corresponding each to its
wavelength used for DWDM communications [2] [5]. GIs are also referred to as
“hubs” later in the paper.

Figure 3 shows an illustrative example of a hybrid-NoC architecture with one
core running a malicious program. The program triggers the infected GI, where
the MR modulators and DB are found, and they become malicious in the network.
The Trojan-infected GI gets activated under a specific condition, manipulating
light and exploiting inherent vulnerabilities in the optical interconnect [5].
In following sections, we explore how these hardware Trojans can insert a
strong jamming signal to commence a gain competition attack in ONoCs.

###  II-B Related Work

There are several research efforts on exploring ONoC security vulnerabilities
and developing effective countermeasures. Many of these vulnerabilities stem
from possible Hardware Trojans (HT) in the network and the manipulation of
MRs. Chittamuru et al. [7] explored a situation where an HT is capable of
manipulating the tuning circuit and can tune MRs to initiate a snooping
attack. The paper proposes an encryption scheme where keys are based on the
individual process variation profiles for each group of MR interfaces. Attacks
that pose serious threats to the performance of electrical NoCs, such as
blackhole attacks, sinkhole attacks, and flooding attacks, were shown for
ONoCs as well. Xie et al. [3] analyzed BER and OSNR of photonic data
transmission in ONoCs. The crosstalk noise is generated when a small amount of
the power from one signal interferes with other signals being used in the
optical network. The noise leads to increased BER and limits the scalability
of ONoCs due to degraded QoS. Xie et al. proposed a novel compact high-SNR
optical router that improves the maximum size of mesh-base ONoC to 8$\times$8
network for BER of $10^{-3}$. Later in our paper, we utilize the range of BER
values given for reliable transmission in [3] to model the proposed gain
competition attack (see Section III-A). Other works also have shown the use of
EDFAs [14] [15] and SOAs [16] in optical NoCs. The effect of introduced noise
on OSNR drop and mitigation of the drop using the optimized placement of SOA
in hybrid-NoC is shown in [17]. Jang et al. [17] show the effect of introduced
noise on SNR drop and how optimized placement of SOA can mitigate this problem
in hybrid-NoC.

There are a few efforts to explore gain competition attacks in the computer
networks domain [10] [18], the attack has not been explored in the context of
optical on-chip communication. To the best of our knowledge, this paper makes
the first attempt to investigate gain competition attacks as a security threat
in ONoC-based SoCs.

Figure 4: A 4$\times$4 hybrid-NoC with 4 optical hubs.

##  III Gain Competition Attack

The gain competition attack utilizes the principle of gain competition that
occurs when amplifying optical signals to deny or lower the quality of service
in an optical network. In a gain competition attack, a malicious component
such as HT-infected GI injects a powerful signal (usually 10 to 20dB stronger
than other signals) of a wavelength different from the legitimate signals but
still in the pass band of the amplifier. After the malicious signal is
injected, it passes through an amplifier. The amplifier cannot distinguish
between the attacking and legitimate signals. Therefore, it amplifies all the
signal that passes through it. The attacking signal will receive more gain
than the legitimate signals because it receives protons proportional to its
power level [2]. This will result in decreased OSNR and, consequently,
increased BER.

There are several factors, such as the power of the malicious signal, the
wavelength distance with legitimate signal, channel spacing, the modulation
speed, and the MR quality, on which gain competition attack depends. However,
in our proposed attack model, we consider two important factors, the strength
of the malicious signal and the distance of the maliciously injected signals
from the legitimate signals, while inserting HT at the GI. The closer the
attacker’s wavelength is to the legitimate wavelengths, the more detrimental
the attack will be on the network. The same goes for the strength of the
attacking signal. Note that a stronger attacking signal means more gain will
be robbed at the amplifier, and the OSNR drop will be more severe [18] [19].

The effectiveness of the gain competition attack increases when optical
signals have to pass through multiple amplifiers, as the OSNR deteriorates at
each amplifier. There are differences in each wavelength’s available gain
growth at each amplifier. If an attacker’s injected signal goes undetected,
this cascading amplifier effect can wreak havoc on an interconnection network.
For larger ONoCs, where the signal passes through multiple amplifiers, this
can be a major problem [19].

Figure 5: A 8$\times$8 hybrid-NoC with 4 optical hubs.

###  III-A Modeling Gain Competition Attacks

We model a gain competition attack on hybrid-NoC using a widely-used Noxim
[12], cycle-accurate SystemC NoC Simulator. For simplicity, we treat the
wireless hubs in Noxim as optical hubs/GIs, assuming the performance to be
comparable. In our model, a hub connects a cluster of cores to the photonic
links necessary for optical data transmission [2] [7]. These hubs contain an
array of MRs to send wavelength-multiplexed signals and an array of DBs to
filter each wavelength and drop them on their respective photodetectors.

To emulate a gain competition attack, we assume one hub ($H_{1}$), as shown in
Figure 4 and 5, can maliciously inject wavelengths and decrease the QoS of a
particular waveguide. IP cores that are connected to this hub can also
initiate an attack if they are running some malicious software component. As
shown in Figure 4, $H_{1}$ is under the control of an attacker and is able to
inject wavelengths strong enough to cause gain competition. This means that
the BER of the optical transmission from $H_{1}$ to $H_{2}$ would be
significantly higher than $H_{0}$ to $H_{1}$. To model this, we modified Noxim
and implemented a wireless BER parameter with the ability to apply it to
specific hub-to-hub connections. The attacking wavelengths used by the
attacker in $H_{1}$ would be the ones that are used for communication within
the network but not for the particular instance of data transmission when the
attack occurs. This makes the attack more covert and feasible than wavelengths
unsupported by the on-chip optical components (i.e., laser, MRs, etc.). We see
legitimate wavelengths used for gain competition attacks in [20] as well.

Figure 6: GAT (flits/cycle) for different BER values in 4$\times$4 hybrid-NoC

Figure 7: Throughput for different BER values in 4$\times$4 hybrid-NoC

We set the wireless BER to $10^{-3}$ to model a hub-to-hub connection that has
undergone a gain competition attack. This means that $1$ in every $1000$ bits
is erroneous, causing any packet containing an erroneous bit to be
retransmitted. We adapt the range of BER values given for reliable
transmission in [3] for our proposed attack modeling. For reliable optical
transmission, the BER has to be in the range of $10^{-9}$ to $10^{-3}$. Our
model uses packet retransmission probability to analyze the effect of the gain
competition attack on NoC. The packet retransmission probability
($P_{retransmit}$) can be calculated as $BER$ $\times$ $f$ $\times$ $p$, where
$f$ is flit size and $p$ is packet size in flits. For example, with a BER of
$10^{-3}$, a flit size of $32$ bits, and a packet size of $8$ flits,
$P_{retransmit}$ $=$ $10^{-3}$ $\times$ $32$ $\times$ $8$ $=$ $0.256$. Roughly
one out of four packets must be retransmitted on the compromised optical
connection. When packet retransmission is necessary, we model a delay
equivalent to one round trip period to account for the transmission delay of
the failed acknowledgment back to the source and retransmit the packet.

Our framework can model gain competition attacks on various NoC configurations
and network topologies executing diverse traffic patterns as discussed in
Section IV. For example, Figure 5 shows the placement of hubs in an
$8$$\times$$8$ mesh hybrid-NoC with $64$ cores. The number of optical hubs
stayed the same as the $4$$\times$$4$ network in Figure 4, but they are placed
in the center of each of the four $16-core$ quadrants, where each core is at
most three hops away from the optical hub. Note that such a hybrid model
introduces interesting routing considerations. For example, a communication
between the $Node_{1}$ and $Node_{55}$ can utilize the path:
$Node_{1}\rightarrow Node_{9}\rightarrow H_{0}\rightarrow H_{2}\rightarrow
Node_{54}\rightarrow Node_{55}$. However, a communication between the
$Node_{1}$ and $Node_{6}$ have many alternatives including only electrical
($Node_{1}\rightarrow Node_{2}\rightarrow Node_{3}\rightarrow
Node_{4}\rightarrow Node_{5}\rightarrow Node_{6}$) and hybrid
($Node_{1}\rightarrow Node_{9}\rightarrow H_{0}\rightarrow H_{1}\rightarrow
Node_{14}\rightarrow Node_{6}$). Our modeling framework will enable designers
to explore different routing protocols with varying optical hub placements
under conflicting design constraints, including security, energy efficiency,
and QoS.

##  IV Experimental Evaluation

Figure 8: Total energy usage ($\mu J$) for different BER values in 4$\times$4
hybrid-NoC

Figure 9: GAD (cycles) for different BER values in 4$\times$4 hybrid-NoC

Figure 10: Throughput (flits/cycle/IP) for different BER values for four
different topologies in 4$\times$4 hybrid-NoC.

Figure 11: GAD (cycles) for different BER values for four different topologies
in 4$\times$4 hybrid-NoC.

We demonstrate the applicability and scalability of our proposed attack on
Noxim Simulator [12], as described in Section III-A. We evaluate the attack on
a 4$\times$4 and an 8$\times$8 hybrid-NoC with $4$ optical hubs as shown in
Figure 4 and Figure 5, respectively. In the network, all the routers are
divided into 4 groups. Within each group, the routers are electrically
connected using mesh topology, and each router in the group is connected to a
different group through an optical link. We simulated the model using four
synthetic traffic patterns Random, BitReversal, Shuffle, and random with a
HotSpot on $Node_{3}$, with a fixed injection rate of 0.005 flits/cycle/node.
We adopt the bit-error rate given in [3] to study the impact of the proposed
attack for different BER values in the range of $10^{-9}$ to $3.5\times
10^{-3}$. We have also shown the behavior of the attack on a variety of
network topologies such as Mesh, Butterfly, Baseline [21], and Omega. Note
that we have used hybrid-NoC to evaluate the proposed attack. However, the
attack is equally applicable to Optical NoC.

For simplicity, in our experiment, we demonstrate the effect of the attack on
one wavelength, as opposed to a wavelength-division-multiplexed network, where
multiple wavelengths are used for data transmission. Note that gain
competition attacks can affect many wavelengths simultaneously and cause DoS
attack. The magnitude of the effect on other wavelengths depends on the
strength of the attacking signal and the difference in wavelengths between the
attacker and the legitimate signals [19].

###  IV-A Results using 4$\times$4 Mesh hybrid-NoC

In this section, we analyze the impact of gain competition attacks on
throughput, energy consumption, and Global Average Delay (GAD). Figure 7 shows
the effect of the attack on Global Average Throughput (GAT). GAT represents
the successful transmission of flits-per-cycle at the destination nodes. As
shown in the graph, performance decrease results from the packet drops due to
the compromised optical link with an elevated BER. This leads to packet
retransmission and a decrease in network throughput. Across all types of
traffic, the network throughput decreased by an average of $68$%, when
comparing the least BER ($10^{-9}$) to the maximum BER ($3.5\times 10^{-3}$)
in our evaluation. Among all four synthetic traffic patterns, BitReversal
showed the least throughput degradation ($58$%), whereas HotSpot suffered from
maximum decrease ($77$%) due to traffic hotspot generated at $Node_{3}$.
$Node_{3}$ resides in the corner and far away from other nodes, making it a
prime candidate for optical transmission (c.f. Figure 4). In traffic patterns,
BitReversal and Shuffle, the number of packets traversing through the
malicious link is significantly low compared to other traffic patterns. Unlike
HotSpot, this reduces the cases of packet retransmission. Figure 7 shows the
average throughput per core in flits/cycle/IP. It shows a trend similar to
that of Figure 7 except being on a per-core scale, demonstrating that
individual cores also experience degraded performance.

Figure 9 shows the effect of the gain competition attack on the NoC’s energy
consumption. As the BER increases, the energy of the network consumption also
increases ($15$%). This can be attributed to the energy required to retransmit
erroneous packets caused by the gain competition attack. We use the default
energy settings in Noxim to generate these results.

Figure 9 shows the global average delay of flits in the network. The delay is
expected to be higher during a gain competition attack. Flits transmitted on
compromised waveguides are more likely to have to be retransmitted, causing
other flits to be delayed. One outlier in this data set is the HotSpot traffic
pattern in the worst BER scenario. The HotSpot traffic pattern makes
$Node_{3}$ a very highly trafficked node for higher BER values, and therefore,
the global average delay is $6.4$ times higher in BER $3.5\times 10^{-3}$ over
$10^{-9}$. The optical waveguides sending data to and around $Node_{3}$ (c.f.
Figure 4) become highly congested in the most extreme attack scenario. With
such a high BER, packets get retransmitted frequently, and the whole network
becomes congested, as flits must wait to be delivered for an alarmingly long
time.

###  IV-B Analyzing the effect on different network topologies

Figure 12: Throughput for different BER values in 8$\times$8 hybrid-NoC.

Figure 13: GAD (cycles) for different BER values in 8$\times$8 hybrid-NoC.

The previous section explored different traffic patterns on a 4$\times$4 mesh
topology. Here, we explore the effect of gain competition attacks on the
average throughput and delay for different network topologies. Figure 11 shows
the global average delay on Mesh, Butterfly, Baseline [21] and Omega topology
with Random traffic pattern. As we can see, a trend similar to Figure 9, where
the global average delay is higher for the higher bit error rates, is found on
the right side of the graph. Similar behavior is observed for average
throughput on four topologies (shown in Figure 11). This demonstrates that our
model is applicable across different NoC topologies.

###  IV-C Analyzing the effect on a bigger network

As shown in Figure 13 and Figure 13, we examine the average throughput per
core and average delay for 8$\times$8 mesh topology (c.f. Figure 5). The
results show the trend of network throughput decreasing as BER increases. The
resulting difference for each traffic pattern is more pronounced on the
8$\times$8 setup than on the 4$\times$4\. As expected, the throughput is close
to the case when there is no attack and the BER value is zero. The global
average delay (GAD) increases with the BER value. Similar to the 4$\times$4
setup, GAD is too high for HotSpot traffic for all BER values. In the bigger
network, the congestion at $Node_{3}$ is much higher even at the low BER
values. This demonstrates that our gain competition attack modeling is
applicable across different NoC configurations and traffic patterns.

##  V Attack Detection and Mitigation

In this section, we discuss a few potential gain competition attack detection
and mitigation techniques.

Attack Detection: Attack detection involves three steps: recognition,
classification and localization, and alarm generation [22]. We explore the
suitability of attack detection from the computer networks domain in ONoC
[19]. Optical Spectrum Analyzer (OSA) provides a more detailed diagnosis than
typical wideband power detection. It can detect a change in the shape of the
optical spectrum used for optical data transfer. OSA measures the individual
wavelengths and amplitudes and generates an alarm on attack detection. Optical
spectrum analysis is a statistical method and requires averaging over time,
making it slower than other methods and potentially unable to detect
infrequent degradation of the signal [19] [22]. In [23], experiments were done
with an on-chip OSA device that effectively measured the amplitude of light
waves respective to their wavelengths. This makes OSA a strong option for
defending against gain competition attacks in ONoCs.

Another statistical method discussed in [24] measures the BER of optical
transmissions over a given period using a piece of hardware called a $BER-
Tester$. This method effectively detects attacks when the quality of an
optical transmission line is degrading due to elevated BER. However, it
requires averaging and comparison, which can delay attack detection.
Monitoring the dropped/retransmitted packets in the network and comparing them
to the threshold can eliminate the hardware overhead involved with a $BER-
Tester$ in ONoC.

Pilot tones are another way to detect the occurrence of a gain competition
attack. They are often out-of-band signals that travel along the same paths as
communication signals but are distinguishable. When it passes through the same
EDFA as the legitimate communication signals, the pilot tones’ quality will be
degraded, and the attack can be detected by monitoring the degradation. This
method may fail if the OSNR is not degraded enough to reach the detection
threshold [19] [22].

Potential Countermeasures: Once an attack is detected, several responses can
happen. The direct response to a gain competition attack is to find and
eliminate the attacking wavelength. OSA can be used to detect the attack, and
measures can be taken to eliminate the wavelength interfering with legitimate
signals. If the above response is not feasible, traffic can be rerouted via
unaffected parts of the on-chip network. In our example (Figure 4), there is
an option to send traffic along the electrical wires in the network. Sending
packets through the electrical route may cause congestion and be slower;
however, this may be a good option if the BER is too high in the optical
waveguide.

Another way to route packets around a gain competition attack is to send
packets on wavelengths that are far away from the jamming signal’s wavelength.
Several experiments have shown that the BER is lower for data transmission on
wavelengths further away from the attacker’s wavelength in a gain competition
attack scenario [18] [19]. This requires a transmission spectrum large enough
to contain wavelengths that are adequately far from the attacker’s wavelength
to have a BER acceptable for optical transmission.

The variant of the $BER-Tester$ approach that uses thresholds appears to be
the best fit for our threat model, considering the resource-constrained nature
of ONoC. However, it is worth noting that the most beneficial detection method
and actions can depend on the architecture and threat models for other
instances.

##  VI Conclusion

Optical network-on-chip architectures provide various advantages over its
electrical counterpart. However, optical NoCs face various security
challenges. In this paper, we investigated gain competition attacks in optical
NoCs. Specifically, we developed a method to model a gain competition attack
using optical bit error rate. We explored the impact of a gain competition
attack using several optical NoC configurations, traffic patterns and
topologies. Our experiments using different NoC topologies and diverse traffic
patterns demonstrated that gain competition attacks present a serious threat
to optical NoCs. We proposed several promising attack detection as well as
attack mitigation techniques that would enable design of trustworthy optical
NoCs.

## References

  * [1] W. J. Dally and B. Towles, “Route packets, not wires: on-chip inteconnection networks,” in _DAC_ , 2001.
  * [2] S. Pasricha and M. Nikdast, “A Survey of Silicon Photonics for Energy Efficient Manycore Computing,” _IEEE Design & Test_, 2020.
  * [3] Y. Xie _et al._ , “Crosstalk noise and bit error rate analysis for optical network-on-chip,” in _DAC_. IEEE, 2010.
  * [4] S. Charles _et al._ , “Real-Time Detection and Localization of Distributed DoS Attacks in NoC-Based SoCs,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 39, no. 12, 2020\.
  * [5] S. V. R. Chittamuru _et al._ , “Exploiting Process Variations to Secure Photonic NoC Architectures from Snooping Attacks,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 2020.
  * [6] M. J. Connelly, _Semiconductor optical amplifiers_. Springer Science & Business Media, 2007.
  * [7] S. V. R. Chittamuru _et al._ , “SOTERIA: exploiting process variations to enhance hardware security with photonic NoC architectures,” in _DAC_ , 2018\.
  * [8] P. Grani and S. Bartolini, “Design Options for Optical Ring Interconnect in Future Client Devices,” _JETCSHT infected_ , vol. 10, no. 4, 2014.
  * [9] M. Furdek, M. Bosiljevac, N. Skorin-Kapov, and Z. Šipuš, “Gain competition in optical amplifiers: a case study,” in _The 33rd International Convention MIPRO_. IEEE, 2010, pp. 467–472.
  * [10] T. Deng and S. Subramaniam, “Analysis of optical amplifier gain competition attack in a point-to-point WDM link,” in _OptiComm 2002: Optical Networking and Communications_ , vol. 4874, 2002.
  * [11] P. Guo _et al._ , “Potential Threats and Possible Countermeasures for Photonic Network-on-Chip,” _IEEE Communications Magazine_ , vol. 58, no. 9, pp. 48–53, 2020.
  * [12] V. Catania _et al._ , “Noxim: An open, extensible and cycle-accurate network on chip simulator,” in _IEEE 26th international conference on application-specific systems, architectures and processors_ , 2015.
  * [13] Y. Pan _et al._ , “Flexishare: Channel sharing for an energy-efficient nanophotonic crossbar,” in _HPCA_. IEEE, 2010.
  * [14] B. G. Lee _et al._ , “Multi-wavelength message routing in a non-blocking four-port bidirectional switch fabric for silicon photonic networks-on-chip,” in _Optical Fiber Communication Conference_ , 2009.
  * [15] R. Ji _et al._ , “Microring-resonator-based four-port optical router for photonic networks-on-chip,” _Optics express_ , vol. 19, no. 20, 2011.
  * [16] I. G. Thakkar _et al._ , “Run-time laser power management in photonic NoCs with on-chip semiconductor optical amplifiers,” in _International Symposium on Networks-on-Chip_. IEEE, 2016\.
  * [17] J. Y. Jang _et al._ , “Power and signal-to-noise ratio optimization in mesh-based hybrid optical network-on-chip using semiconductor optical amplifiers,” _Applied Sciences_ , vol. 9, no. 6, 2019.
  * [18] L. Xiaomin _et al._ , “Implementation and Analysis of EDFA Gain Competition Attack in Optical Network,” in _ISDEA_. IEEE, 2015.
  * [19] M. Furdek and N. Skorin-Kapov, “Physical-layer attacks in transparent optical networks,” _Optical Communications Systems_ , vol. 3, 2012.
  * [20] R. Rejeb _et al._ , “Fault and attack management in all-optical networks,” _IEEE Communications Magazine_ , vol. 44, no. 11, 2006.
  * [21] E. Lu and S. Zheng, “Parallel routing and wavelength assignment for optical multistage interconnection networks,” in _ICPP_. IEEE, 2004.
  * [22] M. Medard _et al._ , “Security issues in all-optical networks,” _IEEE network_ , vol. 11, no. 3, 1997.
  * [23] Y. Li _et al._ , “Integrated on-chip C-band optical spectrum analyzer using dual-ring resonators,” in _CLEO: Science and Innovations_ , 2015.
  * [24] P. Saengudomlert, “Analysis and detection of jamming attacks in an all-optical network,” Ph.D. dissertation, MIT, 1998.


# Light Curves and Event Rates of Axion Instability Supernovae

Kanji Mori Research Institute of Stellar Explosive Phenomena, Fukuoka
University,
8-19-1 Nanakuma, Jonan-ku, Fukuoka-shi, Fukuoka 814-0180, Japan
<EMAIL_ADDRESS>Takashi J. Moriya National Astronomical Observatory
of Japan, 2-21-1 Osawa, Mitaka, Tokyo 181-8588, Japan  School of Physics and
Astronomy, Faculty of Science, Monash University, Clayton, Victoria 3800,
Australia  Tomoya Takiwaki National Astronomical Observatory of Japan, 2-21-1
Osawa, Mitaka, Tokyo 181-8588, Japan  Kei Kotake Research Institute of Stellar
Explosive Phenomena, Fukuoka University,
8-19-1 Nanakuma, Jonan-ku, Fukuoka-shi, Fukuoka 814-0180, Japan  Department of
Applied Physics, Faculty of Science, Fukuoka University,
8-19-1 Nanakuma, Jonan-ku, Fukuoka-shi, Fukuoka 814-0180, Japan  Shunsaku
Horiuchi Center for Neutrino Physics, Department of Physics, Virginia Tech,
Blacksburg, VA 24061, USA  Kavli IPMU (WPI), UTIAS, The University of Tokyo,
Kashiwa, Chiba 277-8583, Japan  Sergei I. Blinnikov National Research Center
“Kurchatov Institute”, 123182 Moscow, Russia  Kavli IPMU (WPI), UTIAS, The
University of Tokyo, Kashiwa, Chiba 277-8583, Japan

###### Abstract

It was recently proposed that exotic particles can trigger a new stellar
instability which is analogous to the $e^{-}e^{+}$ pair instability if they
are produced and reach equilibrium in the stellar plasma. In this study, we
construct axion instability supernova (AISN) models caused by the new
instability to predict their observational signatures. We focus on heavy
axion-like particles (ALPs) with masses of $\sim 400$ keV–$2$ MeV and coupling
with photons of $g_{a\gamma}\sim 10^{-5}$ GeV-1. It is found that the 56Ni
mass and the explosion energy are significantly increased by ALPs for a fixed
stellar mass. As a result, the peak times of the light curves of AISNe occur
earlier than those of standard pair-instability supernovae by 10–20 days when
the ALP mass is equal to the electron mass. Also, the event rate of AISNe is
$1.7$–$2.6$ times higher than that of pair-instability supernovae, depending
on the high mass cutoff of the initial mass function.

Stellar evolution (1599) — Supernovae (1668)

††software: MESA (Paxton et al., 2011, 2013, 2015, 2018, 2019), Reproduction
Package for the Paper “Axion Instability Supernovae” (Sakstein et al., 2022),
STELLA (Blinnikov et al., 1998, 2000, 2006)

##  1 Introduction

The equation of state (EoS) of plasma determines stellar structure and
evolution. In the hot plasma with temperatures $T\gtrsim 10^{9}$ K and the
densities $\rho<10^{6}$ g cm-3, the $e^{-}e^{+}$ pair-creation takes place and
the EoS becomes softer if photons are in thermodynamic equilibrium with the
plasma. As a result, in the carbon-oxygen cores of very massive stars with
initial mass of $M_{\mathrm{init}}\sim 150$–$260M_{\odot}$, the adiabatic
index $\Gamma_{1}$ becomes lower than 4/3 and the star becomes dynamically
unstable. The core then contracts and explosive oxygen burning releases a
large amount of energy to disrupt the whole star (Bisnovatyi-Kogan & Kazhdan,
1967; Rakavy & Shaviv, 1967; Barkat et al., 1967; Fraley, 1968). The stellar
explosion caused by the $e^{-}e^{+}$ pair-creation is called a pair
instability supernova (PISN).

The extremely hot environment in PISNe could be used as a laboratory to probe
fundamental physical processes such as new particles (Croon et al., 2020;
Sakstein et al., 2020, 2022), the 12C$(\alpha,\gamma)^{16}$O reaction rate
(Takahashi, 2018; Farmer et al., 2020), and relativistic plasma screening
(Famiano et al., 2022). These studies were motivated by the recent
gravitational-wave detection of black hole mergers (e.g., The LIGO Scientific
Collaboration et al., 2021). When the stellar mass is $M_{\mathrm{init}}\sim
80$–$150M_{\odot}$, the star is not totally disrupted but a large part of its
mass can be ejected by the pulsational pair instability. As a result, the so-
called pair-instability mass gap of black holes is formed (e.g., Marchant et
al., 2016; Woosley, 2017; Leung et al., 2019; Farmer et al., 2019; Tanikawa et
al., 2021; Kinugawa et al., 2021). Because the position of the lower edge of
the mass gap could be estimated from a large sample of black hole mergers, it
is possible to compare the observations and stellar models to obtain
information on fundamental physics.

Recently, Sakstein et al. (2022) developed stellar models which consider new
bosons beyond the Standard Model. If the new boson is as massive as or lighter
than $\sim 2m_{e}$, where $m_{e}$ is the electron mass, and is strongly
coupled with the plasma, it can be produced in PISNe and reach equilibrium. In
such a case, the EoS is softened and a new stellar instability can be induced.
Sakstein et al. (2022) focused on the pair-instability mass gap and found that
the lower edge of the mass gap decreases by $\sim 10M_{\odot}$ when the mass
of the new particle is equal to $m_{e}$. Although their method is applicable
to any bosons, they focused on heavy axion-like particles (ALPs) that interact
with photons. They coined the term “axion instability supernova” (AISN) for
the new transient that is induced by ALPs.

PISNe and pulsational PISNe have been linked to observed superluminous
supernovae such as SN 2007bi (Gal-Yam & Leonard, 2009), OGLE14-073 (Kozyreva
et al., 2018), and SN 2006gy (Woosley et al., 2007). Also, PISNe are a target
of next-generation near-infrared telescopes such as the Nancy Grace Roman
Space Telescope (Moriya et al., 2022a), Euclid space telescope (Moriya et al.,
2022b; Tanikawa et al., 2022), and James Webb Space Telescope (Hartwig et al.,
2018; Regős et al., 2020). These observation may discover PISN-like events and
enable one to probe their nature. It is hence important to predict observable
quantities beforehand.

Heavy ALPs which couple with photons have been constrained by beam dump
experiments (e.g., Dolan et al., 2017) and astrophysical phenomena including
horizontal branch stars (Carenza et al., 2020; Lucente et al., 2022) and core-
collapse supernovae (Jaeckel et al., 2018; Lucente et al., 2020; Mori et al.,
2022; Caputo et al., 2022a, b). There is an interesting region at $m_{a}\sim
400$ keV–$2$ MeV and $g_{a\gamma}\sim 10^{-5}$ GeV-1 in the ALP parameter
space, where $m_{a}$ is the ALP mass and $g_{a\gamma}$ is the coupling
constant between ALPs and photons. This region is called the “cosmological
triangle” because only cosmological argument has excluded it. Thus, it is
desirable to access this region with independent astrophysical argument as
well. In the cosmological triangle, the ALP-photon coupling is so strong that
ALPs are confined in the stellar core. Since ALPs in this region reach
equilibrium with the stellar plasma, they can affect the EoS and lead to AISNe
(Sakstein et al., 2022). In this paper, we explore observable the signatures
of AISNe caused by ALPs in the cosmological triangle.

This paper is organized as follows. In Section 2, we describe the setup of our
PISN and AISN models. In Section 3, we show the 56Ni mass, the explosion
energy, the light curves, and the event rates inferred from the models. In
Section 4, the future detection prospect is discussed and the paper is
concluded.

##  2 Method

We use Modules for Experiments in Stellar Astrophysics (MESA; Paxton et al.,
2011, 2013, 2015, 2018, 2019) version 12778 to calculate our one-dimensional
stellar models. Input parameters such as the mixing length and overshooting
follow the prescription in Sakstein et al. (2022). Although it solves
hydrostatic evolution for almost all stages of the stellar life, MESA is
capable of switching to the HLLC solver to simulate hydrodynamical evolution
of shocks and pulsations (Paxton et al., 2018). In our models, the
hydrodynamical evolution is solved during the (pulsational) pair instability.

We start our simulations from helium stars with different masses between
$46M_{\odot}$ and $139M_{\odot}$ and the metallicity of $Z=10^{-5}$. We then
follow their evolution until AISNe or core collapse and calculate observable
quantities including the 56Ni mass and event rates. Also, we calculate models
that start from zero-age main sequence (ZAMS) stars with hydrogen envelopes
and follow their evolution until central hydrogen depletion to obtain the
relation between the ZAMS mass $M_{\mathrm{init}}$ and the helium core mass
$M_{\mathrm{He}}$. This relation enables us to calculate the event rates.

When new bosons are produced and trapped in the stellar plasma, the EoS is
modified. The pressure $P_{a}$, density $\rho_{a}$, the internal energy
$u_{a}$, and the specific entropy $s_{a}$ induced by ALPs are given as

$\displaystyle P_{a}$ $\displaystyle=$
$\displaystyle\frac{1}{2}m_{a}c^{2}CH_{1},$ (1) $\displaystyle\rho_{a}$
$\displaystyle=$ $\displaystyle\frac{1}{2}m_{a}CH_{2},$ (2) $\displaystyle
u_{a}$ $\displaystyle=$ $\displaystyle\frac{1}{2}m_{a}c^{2}H_{3},$ (3)
$\displaystyle s_{a}$ $\displaystyle=$
$\displaystyle\frac{kC\beta}{2\rho}(H_{1}+H_{3}),$ (4)

where $k$ is the Boltzmann constant, $c$ is the speed of light,
$C=(m_{a}c/\hbar)^{3}/\pi^{2}$, $\hbar$ is the Planck constant,
$\beta=m_{a}c^{2}/kT$,

$\displaystyle H_{1}$ $\displaystyle=$
$\displaystyle\int^{\infty}_{\beta}G\left(\frac{\epsilon}{\beta}\right)B(\epsilon)\frac{d\epsilon}{\beta},$
(5) $\displaystyle H_{2}$ $\displaystyle=$
$\displaystyle\int^{\infty}_{\beta}G^{\prime}\left(\frac{\epsilon}{\beta}\right)B(\epsilon)\frac{d\epsilon}{\beta},$
(6) $\displaystyle H_{3}$ $\displaystyle=$
$\displaystyle\int^{\infty}_{\beta}\epsilon
G^{\prime}\left(\frac{\epsilon}{\beta}\right)B(\epsilon)\frac{d\epsilon}{\beta^{2}},$
(7)

$B(\epsilon)=(e^{\epsilon}-1)^{-1}$, and $G(x)=(x^{2}-1)^{\frac{3}{2}}/3$. The
modification to other thermodynamic quantities stems from these terms as
tabulated in Sakstein et al. (2022).

In this study, we focus on photophilic ALPs in the cosmological triangle. We
adopt two ALP masses of $f=m_{a}/2m_{e}=0.5$ and 2. Here, the ALP mass is
normalized to $2m_{e}$ because $m_{a}\lesssim 2m_{e}$ is the critical
condition for the new stellar instability. Since the cosmological triangle is
located at $m_{a}\sim 400$ keV–$2$ MeV (Lucente et al., 2022), the two cases
approximately correspond to the lower and the upper limits of $m_{a}$. Also,
we calculate standard PISN models for comparison.

##  3 Results

Table 1: The stellar models developed in this study. In the table, $f=m_{a}/2m_{e}$ is the ALP mass, $M_{\mathrm{init}}$ is the intial stellar mass, $M_{\mathrm{He}}$ is the helium core mass, $M_{\mathrm{BH}}$ is the black hole mass, and $M_{\mathrm{Ni}}$ is the 56Ni mass in the ejecta. The first column is filled with “$-$” for the standard models without ALPs. “PAISN” and “PPISN” in the seventh column stand for the pulsational AISN and the pulsational PISN, respectively. $M_{\mathrm{Ni}}$ and $E_{\mathrm{exp}}$ for PPISNe and PAISNe are not shown because they can cause multiple pulses.  $f$ | $M_{\mathrm{init}}/M_{\odot}$ | $M_{\mathrm{He}}/M_{\odot}$ | $M_{\mathrm{BH}}/M_{\odot}$ | $M_{\mathrm{Ni}}/M_{\odot}$ |  $E_{\mathrm{exp}}$ [$10^{51}$ erg] | Fate
---|---|---|---|---|---|---
0.5 | 100 | 46 | 26.2 |  |  | PAISN
0.5 | 110 | 51 | 0 | 1.07$\times 10^{-3}$ | 4.6 | AISN
0.5 | 120 | 56 | 0 | 2.01$\times 10^{-2}$ | 9.0 | AISN
0.5 | 130 | 59 | 0 | 8.24$\times 10^{-2}$ | 13 | AISN
0.5 | 140 | 67 | 0 | 4.81$\times 10^{-1}$ | 23 | AISN
0.5 | 150 | 72 | 0 | 1.58 | 29 | AISN
0.5 | 160 | 77 | 0 | 4.16 | 35 | AISN
0.5 | 170 | 83 | 0 | 9.36 | 43 | AISN
0.5 | 180 | 88 | 0 | 15.0 | 49 | AISN
0.5 | 190 | 94 | 0 | 23.8 | 57 | AISN
0.5 | 200 | 98 | 0 | 31.3 | 63 | AISN
0.5 | 210 | 102 | 0 | 40.9 | 69 | AISN
2.0 | 140 | 67 | 15.3 |  |  | PAISN
2.0 | 150 | 72 | 0 | 2.20$\times 10^{-2}$ | 7.6 | AISN
2.0 | 160 | 77 | 0 | 7.95$\times 10^{-2}$ | 12 | AISN
2.0 | 170 | 83 | 0 | 3.12$\times 10^{-1}$ | 20 | AISN
2.0 | 180 | 88 | 0 | 8.78$\times 10^{-1}$ | 28 | AISN
2.0 | 190 | 94 | 0 | 2.80 | 36 | AISN
2.0 | 200 | 98 | 0 | 5.21 | 42 | AISN
2.0 | 210 | 102 | 0 | 8.26 | 47 | AISN
2.0 | 220 | 109 | 0 | 15.9 | 58 | AISN
2.0 | 230 | 115 | 0 | 26.2 | 68 | AISN
2.0 | 240 | 122 | 0 | 43.3 | 82 | AISN
$-$ | 140 | 67 | 30.4 |  |  | PPISN
$-$ | 150 | 72 | 0 | 1.71$\times 10^{-2}$ | 7.1 | PISN
$-$ | 160 | 77 | 0 | 6.73$\times 10^{-2}$ | 11 | PISN
$-$ | 170 | 83 | 0 | 2.42$\times 10^{-1}$ | 19 | PISN
$-$ | 180 | 88 | 0 | 6.74$\times 10^{-1}$ | 27 | PISN
$-$ | 190 | 94 | 0 | 2.09 | 35 | PISN
$-$ | 200 | 98 | 0 | 3.97 | 40 | PISN
$-$ | 210 | 102 | 0 | 6.44 | 48 | PISN
$-$ | 220 | 109 | 0 | 12.6 | 54 | PISN
$-$ | 230 | 115 | 0 | 20.1 | 62 | PISN
$-$ | 240 | 122 | 0 | 32.6 | 73 | PISN
$-$ | 250 | 127 | 0 | 43.5 | 83 | PISN

###  3.1 Model Properties

The stellar models developed in this study are listed in Table 1. When the
stellar mass is not sufficiently high, the star experiences the pulsational
pair instability or pulsational axion instability. In these models, the
instability causes stellar pulsations which eject a part of the stellar mass.
As a result, a black hole lighter than $M_{\mathrm{He}}$ is left after the
core collapse. Sakstein et al. (2022) reported that the dynamics of the
pulsation is significantly affected by ALPs and the lower edge of the black
hole mass gap becomes lighter. They also reported that the boundary between
the pulsational instabilities and the total disruption of the star becomes
lighter. In our models, the boundary is at $M_{\mathrm{init}}\approx
145M_{\odot}$ for the standard case and at $M_{\mathrm{init}}\approx
105M_{\odot}$ for the $f=0.5$ case.

On the other hand, when the star is heavy enough, the energy release of oxygen
burning is not sufficient to stop the stellar contraction. In this case, the
total energy of the star is absorbed by photodisintegration. As a result, the
star directly collapses to a black hole (e.g., Takahashi et al., 2016). This
defines the upper edge of the pair-instability mass gap of black holes.
Sakstein et al. (2022) showed that the upper edge becomes lighter if ALPs are
included. In our calculation, the boundary between the total disruption and
the direct collapse is lowered from $M_{\mathrm{init}}\approx 265M_{\odot}$ to
$M_{\mathrm{init}}\approx 215M_{\odot}$ when $f=0.5$ is assumed.

Figure 1: The 56Ni mass $M_{\mathrm{Ni}}$ ejected from AISNe and PISNe as a
function of the helium core mass $M_{\mathrm{He}}$. The solid lines show our
models and the dotted line shows PISN models in Heger & Woosley (2002).
Figure 2: The explosion energies $E_{\mathrm{exp}}$ for AISN and PISN models
as a function of the helium core mass $M_{\mathrm{He}}$. The solid lines show
our models and the dotted line shows PISN models in Heger & Woosley (2002).

Figure 1 shows the ejected 56Ni mass $M_{\mathrm{Ni}}$ of AISNe and PISNe,
which is defined as

$\displaystyle M_{\mathrm{Ni}}=\int X_{{}^{56}\mathrm{Ni}}dM_{r},$ (8)

where $X_{{}^{56}\mathrm{Ni}}$ is the mass fraction of 56Ni and $M_{r}$ is the
mass coordinate. $M_{\mathrm{Ni}}$ increases as a function of the stellar mass
for a fixed $m_{a}$ because the temperature in the stellar core is higher in
heavier stars. This is consistent with previous PISN models (Heger & Woosley,
2002; Kasen et al., 2011; Gilmer et al., 2017; Takahashi et al., 2016, 2018).
On the other hand, when the stellar mass is fixed, $M_{\mathrm{Ni}}$ in AISNe
is significantly larger than that in PISNe. This is because ALPs soften the
EoS and the stellar contraction lasts until the central temperature becomes
higher than standard PISNe. The iron group elements are produced in the
innermost region where the temperature exceeds $\sim 4.5\times 10^{9}$ K
(Takahashi et al., 2018).

Figure 2 shows the explosion energies $E_{\mathrm{exp}}$ for the AISN and PISN
models. As reported in the previous studies (e.g., Heger & Woosley, 2002;
Takahashi et al., 2016), $E_{\mathrm{exp}}$ increases as a linear function of
$M_{\mathrm{He}}$. PISNe and AISNe are more energetic than typical type II
supernovae. Even the lightest PISN model with $M_{\mathrm{He}}=72M_{\odot}$
reaches $E_{\mathrm{exp}}=7.1\times 10^{51}$ erg. Such energetic explosions
are induced by explosive oxygen burning, which is ignited when the central
temperature reaches $\sim 3\times 10^{9}$ K. Since the softened EoS induces
higher temperature in the core, $E_{\mathrm{exp}}$ in AISNe are higher than
the corresponding values in PISNe for a fixed $M_{\mathrm{He}}$.

###  3.2 Light Curves

Figure 3: The bolometric light curve of PISNe (upper panel) and AISNe (lower
panel) with $f=0.5$.

Although confirmed light curves of PISNe have not been observed, they may be
discovered by future observations and provide information on the ejecta mass
$M_{\mathrm{ej}}$, $M_{\mathrm{Ni}}$, and $E_{\mathrm{exp}}$ through the
comparison with the Arnett law (Arnett, 1982) or hydrodynamical simulations
(e.g., Kasen et al., 2011). If the progenitor of PISNe and AISNe is hydrogen-
free, the stellar mass $M_{\mathrm{He}}$ can be estimated from
$M_{\mathrm{ej}}$.

Figure 3 shows the light curves of PISNe and AISNe with $f=0.5$ estimated as
(Arnett, 1982)

$\displaystyle
L(t)=M_{\mathrm{Ni}}e^{-x^{2}}\left((s_{\mathrm{Ni}}-s_{\mathrm{Co}})\int^{x}_{0}2ze^{z^{2}-2zy}dz+\right.$
$\displaystyle\left.s_{\mathrm{Co}}\int
2ze^{z^{2}-2yz+2zs}dz\right)(1-e^{-A_{\gamma}t^{-2}}),$ (9)

where $s_{\mathrm{Ni}}$ and $s_{\mathrm{Co}}$ are the energy generation rates
of the 56Ni and 56Co decays, $x=t/t_{\mathrm{eff}}$,
$y=t_{\mathrm{eff}}/2\tau_{\mathrm{Ni}}$,
$s=t_{\mathrm{eff}}(\tau_{\mathrm{Co}}-\tau_{\mathrm{Ni}})/2\tau_{\mathrm{Ni}}\tau_{\mathrm{Co}}$,
and $A_{\gamma}=(3\kappa_{\gamma}M_{\mathrm{ej}})/(4\pi v^{2})$. Here
$t_{\mathrm{eff}}=\sqrt{2t_{\mathrm{d}}t_{\mathrm{h}}}$ is the effective
diffusion timescale derived from the diffusion timescale $t_{\mathrm{d}}$ and
the dynamical timescale $t_{\mathrm{h}}$, $\tau_{\mathrm{Ni}}$ and
$\tau_{\mathrm{Co}}$ are the 56Ni and 56Co lifetimes, $\kappa_{\gamma}$ is the
$\gamma$-ray opacity, and $v$ is the photospheric velocity.

Figure 4: The peak luminosity $L_{\mathrm{max}}$ and the peak time
$t_{\mathrm{max}}$ for the PISN and AISN models. The color of each point shows
the helium core mass $M_{\mathrm{He}}$.

From Fig. 3, it is seen that the peak luminosity $L_{\mathrm{max}}$ increases
as a function of $M_{\mathrm{He}}$. This is because the luminosity is powered
by 56Ni, which is produced more abundantly in heavier models. Although the
heaviest models can be as luminous as $\sim 10^{44}$ erg s-1, some of lighter
models are less luminous than the typical supernova luminosity $\sim 10^{42}$
erg s-1. The peak time $t_{\mathrm{max}}$ at which the luminosity reaches
$L_{\mathrm{max}}$ is 100–150 days. The luminosity is low in the early days of
$t<t_{\mathrm{max}}$ because the ejecta is optically thick, while it decreases
in the later days because the heating rate becomes lower. The timescale
$t_{\mathrm{max}}$ is determined by the condition $t_{\mathrm{d}}\approx
t_{\mathrm{h}}$ (e.g., Chapter 5 in Branch & Wheeler, 2017). It is easily
shown that this condition leads to the relation
$t_{\mathrm{max}}\propto(M_{\mathrm{ej}}^{3}/E_{\mathrm{exp}})^{4}$.

Since the light curves depend on the supernova properties, they provide
information on $M_{\mathrm{Ni}}$ and $E_{\mathrm{exp}}$ if they are discovered
in future observations. Figure 4 shows $t_{\mathrm{max}}$ and
$L_{\mathrm{max}}$ for PISNe and AISNe. While $L_{\mathrm{max}}$ increases as
a function of $M_{\mathrm{He}}$, $t_{\mathrm{max}}$ is not monotonous because
$t_{\mathrm{max}}\propto(M_{\mathrm{ej}}^{3}/E_{\mathrm{exp}})^{4}$ and both
$M_{\mathrm{ej}}$ and $E_{\mathrm{exp}}$ increase as a function of
$M_{\mathrm{He}}$. It is seen from Fig. 4 that $t_{\mathrm{max}}$ for AISNe
with $f=0.5$ is shorter than that for PISNe. This is because AISNe show larger
$E_{\mathrm{exp}}$ than PISNe, as seen in Fig. 2.

Figure 5: The color measured in $B-V$ of a PISN (upper panel) and an AISN
(lower panel) with $f=0.5$. The stellar mass is fixed to
$M_{\mathrm{He}}=98M_{\odot}$.

Since observational instruments adopt filters which transmit only a specific
frequency band, the bolometric light curves cannot be directly compared to
observations. In order to compare the models with future observations, we
calculated multi-band light curves using a radiation hydrodynamics code STELLA
(Blinnikov et al., 1998, 2000, 2006). In Fig. 5, we show the color measured in
$B-V$. The plot shows that PISNe are redder than AISNe for a fixed stellar
mass. At $t=t_{\mathrm{max}}$, $B-V=1.81$ mag for the PISN and $1.42$ mag for
the AISN. This is because the color is dependent on the ratio between
$M_{\mathrm{Ni}}$ and $M_{\mathrm{He}}$ (Dessart et al., 2012). In the two
models shown in Fig. 5, $M_{\mathrm{He}}$ is fixed but $M_{\mathrm{Ni}}$ is
larger in the AISN model. When $M_{\mathrm{Ni}}$ is higher, heating induced by
the 56Ni decay chain increases the photospheric temperature and leads to bluer
light curves.

###  3.3 Event Rates

Figure 6: The event rate ratio $R$ as a function of the IMF cutoff
$M_{\mathrm{up}}$. Two values of the IMF slope $\alpha=2.35$ and 0 are
adopted.

Because the mass range for AISNe is lighter than that for PISNe, the event
rates for these transients are different. Figure 6 shows the event rate ratio
between AISNe and PISNe, estimated as

$R=\frac{\int^{\min(M_{\mathrm{init,max}}^{\mathrm{ALP}},\;M_{\mathrm{up}})}_{M_{\mathrm{init,min}}^{\mathrm{ALP}}}M^{-\alpha}dM}{\int^{\min(M_{\mathrm{init,max}},\;M_{\mathrm{up}})}_{M_{\mathrm{init,min}}}M^{-\alpha}dM},$
(10)

where $\alpha$ is the slope of the initial mass function (IMF),
$M_{\mathrm{up}}$ is the high mass cutoff of the IMF, and
$M_{\mathrm{init,\;max}}$ $(M_{\mathrm{init,\;min}})$ is the maximum (minimum)
initial mass of PISNe and AISNe. The superscript ALP stands for quantities for
AISNe. Although the typical value for $\alpha$ is 2.35 (Salpeter, 1955),
hydrodynamical simulations for the formation of metal-free stars suggest top-
heavy IMFs (e.g., Susa et al., 2014; Hirano et al., 2014, 2015; Stacy et al.,
2016; Tarumi et al., 2020; Chon et al., 2021). In Fig. 6 we show two IMFs, a
flat IMF with $\alpha=0$ to represent the top-heavy IMFs and the Salpeter IMF
with $\alpha=2.35$. Schneider et al. (2014) analyzed very massive stars in the
stellar cluster R136 and reported that $M_{\mathrm{up}}$ for metal-rich stars
would be in the range $200$–$500M_{\odot}$. As for metal-free stars, stars
heavier than $300M_{\odot}$ were not formed in cosmological simulations
performed by Susa et al. (2014), while simulations in Hirano et al. (2014)
reported the formation of stars as heavy as $\sim 1000M_{\odot}$. Since the
maximum mass for the star formation is highly uncertain, $M_{\mathrm{up}}$ is
adopted as a free parameter in Fig. 6.

Figure 6 shows that the PISN-like event rate is enhanced by a factor of $\sim
1.7$–$2.6$ for $f=0.5$ and the Salpeter IMF. If the ALP mass is larger, the
event rate enhancement is suppressed because the AISN mass range becomes
closer to the PISN case. The event rate ratio is higher with smaller
$M_{\mathrm{up}}$ because the PISN range extends to heavier masses compared
with AISNe and the formation of such heavy stars is prevented by the IMF
cutoff. Although it would be necessary to consider a realistic star formation
history to predict the expected event number for surveys, the number of AISNe
would be systematically higher than PISNe.

###  3.4 Prospects for Constraining the ALP Mass with AISN Light Curves

In Figs. 1 and 2, we saw that the 56Ni mass and the explosion energy change as
a function of $m_{a}$ for a fixed $M_{\mathrm{He}}$. It is thus expected that
estimating $M_{\mathrm{He}}$, $M_{\mathrm{Ni}}$, and $E_{\mathrm{exp}}$ in
future PISNe or AISNe would give a clue to distinguish the two scenarios. As
Fig. 4 shows, they can be estimated from $t_{\mathrm{max}}$ and
$L_{\mathrm{max}}$ of the light curve. Although light curves of PISN-like
events have not been discovered, future observations are planned to find such
events (Hartwig et al., 2018; Regős et al., 2020; Moriya et al., 2022a, b;
Tanikawa et al., 2022). Comparison between the AISN models and observed light
curves would provide a constraint on $m_{a}$.

In our calculations, we assumed that the progenitor is hydrogen-free. If the
helium core is surrounded by a thick envelope, the morphology of the light
curves becomes more complex (e.g., Kasen et al., 2011). In addition, the
envelope makes it impossible to equate $M_{\mathrm{ej}}$ with
$M_{\mathrm{He}}$. In order to extract the information of the hydrogenic
progenitor, it is desirable to perform hydrodynamic modeling of the light
curves. Also, it is notable that the nebular phase spectra of PISNe would
provide information on $M_{\mathrm{Ni}}$ even for the hydrogenic progenitors
(Jerkstrand et al., 2016).

It is known that the evolution of PISNe is sensitive to the
12C$(\alpha,\;\gamma)^{16}$O reaction rate (Takahashi, 2018; Farmer et al.,
2020). In particular, when the rate is higher, the 56Ni production becomes
more efficient. Because the low-energy cross sections of the reaction are
still uncertain, it might be difficult to distinguish the effects of ALPs and
the high reaction rate. Future experiments such as JUNA (Liu et al., 2022) may
reduce the uncertainty in the reaction rate.

##  4 Conclusion

In this study, we explored the 56Ni mass and the event rates of axion
instability supernovae (AISNe). Searches for PISN-like events including AISNe
are being planned using next-generation near-infrared telescopes. Once a
PISN/AISN candidate is discovered, its light curve and spectra would give us
information on the 56Ni mass and the stellar mass. As we saw in Fig. 1, these
quantities can be used to distinguish PISNe and AISNe. Also, as discussed in
Sakstein et al. (2022), the mass distribution of astrophysical black holes
would be affected by heavy ALPs. Optical observations of AISNe and
gravitational-wave observations of black hole binaries would hence provide
complementary methods to probe the nature of ALPs.

An object of interest is the superluminous supernova PTF12dam (Quimby et al.,
2012). It is argued that this object is not a PISN because of its short rise
time (Nicholl et al., 2013). Although this object has a shorter rise time than
our AISN models, it would be worthwhile to perform sensitivity studies on the
12C($\alpha,\;\gamma)^{16}$O reaction rate because a larger rate would lead to
a smaller $M_{\mathrm{ej}}$ for a fixed $M_{\mathrm{Ni}}$ and shorten the
model rise time.

When a star is not massive enough to reach total disruption, it can cause the
pulsational pair or the pulsational axion instabilities (Sakstein et al.,
2022). It would be interesting to explore the models for these cases as well,
because there is a candidate for a pulsational pair instability supernova
(Woosley & Smith, 2022) and more examples would be discovered by future
observations.

Furthermore, nucleosynthesis in PISNe and AISNe may have left traces in
elemental abundances of metal-poor stars. Aoki et al. (2014) reported that a
very metal-poor star SDSS J001820.5–093939.2 shows an elemental composition
that is similar to PISN yields. Although such stars are extremely rare, it is
worthwhile to perform post-process network calculations with a large nuclear
reaction network to predict detailed nucleosynthesis yields of AISNe.

In this calculation, we focused on photophilic ALPs because AISNe provide a
unique way to probe an interesting parameter region called the cosmological
triangle. However, new transients that are similar to PISNe can be induced by
any new bosons if they are tightly coupled with the plasma (Sakstein et al.,
2022). Extending the calculations to various particles would provide a general
way to constrain physics beyond the Standard Model.

The authors thank Jeremy Sakstein and Djuna Croon for fruitful discussion.
K.M. was supported by Research Institute of Stellar Explosive Phenomena
(REISEP) at Fukuoka University and JSPS KAKENHI Grant Number JP21K20369. K.K.
was supported in part by Grants-in-Aid for Scientific Research of the Japan
Society for the Promotion of Science (JSPS, No. JP22H01223), the Ministry of
Education, Science and Culture of Japan (MEXT, Nos. JP17H06364, JP17H06365),
by the REISEP (No. 207002), and JICFuS as “Program for Promoting researches on
the Supercomputer Fugaku” (Toward a unified view of the universe: from large
scale structures to planets, JPMXP1020200109). The work of S.H. is supported
by the U.S. Department of Energy Office of Science under award number DE-
SC0020262, NSF Grant No. AST1908960 and No. PHY-1914409, and JSPS KAKENHI
Grant Number JP22K03630. This work was supported by World Premier
International Research Center Initiative (WPI Initiative), MEXT, Japan. S.B.
is supported by RSCF grant 19-12-00229 in his work on SN light curve
simulations. Numerical computations were performed by the PC cluster at Center
for Computational Astrophysics, National Astronomical Observatory of Japan.

## References

  * Aoki et al. (2014) Aoki, W., Tominaga, N., Beers, T. C., Honda, S., & Lee, Y. S. 2014, Science, 345, 912, doi: 10.1126/science.1252633
  * Arnett (1982) Arnett, W. D. 1982, ApJ, 253, 785, doi: 10.1086/159681
  * Barkat et al. (1967) Barkat, Z., Rakavy, G., & Sack, N. 1967, Phys. Rev. Lett., 18, 379, doi: 10.1103/PhysRevLett.18.379
  * Bisnovatyi-Kogan & Kazhdan (1967) Bisnovatyi-Kogan, G. S., & Kazhdan, Y. M. 1967, Soviet Astronomy, 10, 604
  * Blinnikov et al. (2000) Blinnikov, S., Lundqvist, P., Bartunov, O., Nomoto, K., & Iwamoto, K. 2000, ApJ, 532, 1132, doi: 10.1086/308588
  * Blinnikov et al. (1998) Blinnikov, S. I., Eastman, R., Bartunov, O. S., Popolitov, V. A., & Woosley, S. E. 1998, ApJ, 496, 454, doi: 10.1086/305375
  * Blinnikov et al. (2006) Blinnikov, S. I., Röpke, F. K., Sorokina, E. I., et al. 2006, A&A, 453, 229, doi: 10.1051/0004-6361:20054594
  * Branch & Wheeler (2017) Branch, D., & Wheeler, J. C. 2017, Supernova Explosions, doi: 10.1007/978-3-662-55054-0
  * Caputo et al. (2022a) Caputo, A., Janka, H.-T., Raffelt, G., & Vitagliano, E. 2022a, arXiv e-prints, arXiv:2201.09890.  https://arxiv.org/abs/2201.09890
  * Caputo et al. (2022b) Caputo, A., Raffelt, G., & Vitagliano, E. 2022b, Phys. Rev. D, 105, 035022, doi: 10.1103/PhysRevD.105.035022
  * Carenza et al. (2020) Carenza, P., Straniero, O., Döbrich, B., et al. 2020, Physics Letters B, 809, 135709, doi: 10.1016/j.physletb.2020.135709
  * Chon et al. (2021) Chon, S., Omukai, K., & Schneider, R. 2021, MNRAS, 508, 4175, doi: 10.1093/mnras/stab2497
  * Croon et al. (2020) Croon, D., McDermott, S. D., & Sakstein, J. 2020, Phys. Rev. D, 102, 115024, doi: 10.1103/PhysRevD.102.115024
  * Dessart et al. (2012) Dessart, L., Hillier, D. J., Waldman, R., Livne, E., & Blondin, S. 2012, MNRAS, 426, L76, doi: 10.1111/j.1745-3933.2012.01329.x
  * Dolan et al. (2017) Dolan, M. J., Ferber, T., Hearty, C., Kahlhoefer, F., & Schmidt-Hoberg, K. 2017, Journal of High Energy Physics, 2017, 94, doi: 10.1007/JHEP12(2017)094
  * Famiano et al. (2022) Famiano, M. A., Mori, K., Balantekin, A. B., et al. 2022, A&A, 659, A97, doi: 10.1051/0004-6361/202142433
  * Farmer et al. (2020) Farmer, R., Renzo, M., de Mink, S. E., Fishbach, M., & Justham, S. 2020, ApJ, 902, L36, doi: 10.3847/2041-8213/abbadd
  * Farmer et al. (2019) Farmer, R., Renzo, M., de Mink, S. E., Marchant, P., & Justham, S. 2019, ApJ, 887, 53, doi: 10.3847/1538-4357/ab518b
  * Fraley (1968) Fraley, G. S. 1968, Ap&SS, 2, 96, doi: 10.1007/BF00651498
  * Gal-Yam & Leonard (2009) Gal-Yam, A., & Leonard, D. C. 2009, Nature, 458, 865, doi: 10.1038/nature07934
  * Gilmer et al. (2017) Gilmer, M. S., Kozyreva, A., Hirschi, R., Fröhlich, C., & Yusof, N. 2017, ApJ, 846, 100, doi: 10.3847/1538-4357/aa8461
  * Hartwig et al. (2018) Hartwig, T., Bromm, V., & Loeb, A. 2018, MNRAS, 479, 2202, doi: 10.1093/mnras/sty1576
  * Heger & Woosley (2002) Heger, A., & Woosley, S. E. 2002, ApJ, 567, 532, doi: 10.1086/338487
  * Hirano et al. (2015) Hirano, S., Hosokawa, T., Yoshida, N., Omukai, K., & Yorke, H. W. 2015, MNRAS, 448, 568, doi: 10.1093/mnras/stv044
  * Hirano et al. (2014) Hirano, S., Hosokawa, T., Yoshida, N., et al. 2014, ApJ, 781, 60, doi: 10.1088/0004-637X/781/2/60
  * Jaeckel et al. (2018) Jaeckel, J., Malta, P. C., & Redondo, J. 2018, Phys. Rev. D, 98, 055032, doi: 10.1103/PhysRevD.98.055032
  * Jerkstrand et al. (2016) Jerkstrand, A., Smartt, S. J., & Heger, A. 2016, MNRAS, 455, 3207, doi: 10.1093/mnras/stv2369
  * Kasen et al. (2011) Kasen, D., Woosley, S. E., & Heger, A. 2011, ApJ, 734, 102, doi: 10.1088/0004-637X/734/2/102
  * Kinugawa et al. (2021) Kinugawa, T., Nakamura, T., & Nakano, H. 2021, MNRAS, 501, L49, doi: 10.1093/mnrasl/slaa191
  * Kozyreva et al. (2018) Kozyreva, A., Kromer, M., Noebauer, U. M., & Hirschi, R. 2018, MNRAS, 479, 3106, doi: 10.1093/mnras/sty983
  * Leung et al. (2019) Leung, S.-C., Nomoto, K., & Blinnikov, S. 2019, ApJ, 887, 72, doi: 10.3847/1538-4357/ab4fe5
  * Liu et al. (2022) Liu, W., Li, Z., He, J., et al. 2022, in European Physical Journal Web of Conferences, Vol. 260, European Physical Journal Web of Conferences, 08001, doi: 10.1051/epjconf/202226008001
  * Lucente et al. (2020) Lucente, G., Carenza, P., Fischer, T., Giannotti, M., & Mirizzi, A. 2020, J. Cosmology Astropart. Phys, 2020, 008, doi: 10.1088/1475-7516/2020/12/008
  * Lucente et al. (2022) Lucente, G., Straniero, O., Carenza, P., Giannotti, M., & Mirizzi, A. 2022, arXiv e-prints, arXiv:2203.01336.  https://arxiv.org/abs/2203.01336
  * Marchant et al. (2016) Marchant, P., Langer, N., Podsiadlowski, P., Tauris, T. M., & Moriya, T. J. 2016, A&A, 588, A50, doi: 10.1051/0004-6361/201628133
  * Mori et al. (2022) Mori, K., Takiwaki, T., Kotake, K., & Horiuchi, S. 2022, Phys. Rev. D, 105, 063009, doi: 10.1103/PhysRevD.105.063009
  * Moriya et al. (2022a) Moriya, T. J., Quimby, R. M., & Robertson, B. E. 2022a, ApJ, 925, 211, doi: 10.3847/1538-4357/ac415e
  * Moriya et al. (2022b) Moriya, T. J., Inserra, C., Tanaka, M., et al. 2022b, arXiv e-prints, arXiv:2204.08727.  https://arxiv.org/abs/2204.08727
  * Nicholl et al. (2013) Nicholl, M., Smartt, S. J., Jerkstrand, A., et al. 2013, Nature, 502, 346, doi: 10.1038/nature12569
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3, doi: 10.1088/0067-0049/192/1/3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4, doi: 10.1088/0067-0049/208/1/4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15, doi: 10.1088/0067-0049/220/1/15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34, doi: 10.3847/1538-4365/aaa5a8
  * Paxton et al. (2019) Paxton, B., Smolec, R., Schwab, J., et al. 2019, ApJS, 243, 10, doi: 10.3847/1538-4365/ab2241
  * Quimby et al. (2012) Quimby, R. M., Arcavi, I., Sternberg, A., et al. 2012, The Astronomer’s Telegram, 4121, 1
  * Rakavy & Shaviv (1967) Rakavy, G., & Shaviv, G. 1967, ApJ, 148, 803, doi: 10.1086/149204
  * Regős et al. (2020) Regős, E., Vinkó, J., & Ziegler, B. L. 2020, ApJ, 894, 94, doi: 10.3847/1538-4357/ab8636
  * Sakstein et al. (2022) Sakstein, J., Croon, D., & McDermott, S. D. 2022, Phys. Rev. D, 105, 095038, doi: 10.1103/PhysRevD.105.095038
  * Sakstein et al. (2020) Sakstein, J., Croon, D., McDermott, S. D., Straight, M. C., & Baxter, E. J. 2020, Phys. Rev. Lett., 125, 261105, doi: 10.1103/PhysRevLett.125.261105
  * Salpeter (1955) Salpeter, E. E. 1955, ApJ, 121, 161, doi: 10.1086/145971
  * Schneider et al. (2014) Schneider, F. R. N., Izzard, R. G., de Mink, S. E., et al. 2014, ApJ, 780, 117, doi: 10.1088/0004-637X/780/2/117
  * Stacy et al. (2016) Stacy, A., Bromm, V., & Lee, A. T. 2016, MNRAS, 462, 1307, doi: 10.1093/mnras/stw1728
  * Susa et al. (2014) Susa, H., Hasegawa, K., & Tominaga, N. 2014, ApJ, 792, 32, doi: 10.1088/0004-637X/792/1/32
  * Takahashi (2018) Takahashi, K. 2018, ApJ, 863, 153, doi: 10.3847/1538-4357/aad2d2
  * Takahashi et al. (2018) Takahashi, K., Yoshida, T., & Umeda, H. 2018, ApJ, 857, 111, doi: 10.3847/1538-4357/aab95f
  * Takahashi et al. (2016) Takahashi, K., Yoshida, T., Umeda, H., Sumiyoshi, K., & Yamada, S. 2016, MNRAS, 456, 1320, doi: 10.1093/mnras/stv2649
  * Tanikawa et al. (2022) Tanikawa, A., Moriya, T. J., Tominaga, N., & Yoshida, N. 2022, arXiv e-prints, arXiv:2204.09402.  https://arxiv.org/abs/2204.09402
  * Tanikawa et al. (2021) Tanikawa, A., Susa, H., Yoshida, T., Trani, A. A., & Kinugawa, T. 2021, ApJ, 910, 30, doi: 10.3847/1538-4357/abe40d
  * Tarumi et al. (2020) Tarumi, Y., Hartwig, T., & Magg, M. 2020, ApJ, 897, 58, doi: 10.3847/1538-4357/ab960d
  * The LIGO Scientific Collaboration et al. (2021) The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, et al. 2021, arXiv e-prints, arXiv:2111.03634.  https://arxiv.org/abs/2111.03634
  * Woosley (2017) Woosley, S. E. 2017, ApJ, 836, 244, doi: 10.3847/1538-4357/836/2/244
  * Woosley et al. (2007) Woosley, S. E., Blinnikov, S., & Heger, A. 2007, Nature, 450, 390, doi: 10.1038/nature06333
  * Woosley & Smith (2022) Woosley, S. E., & Smith, N. 2022, arXiv e-prints, arXiv:2205.06386.  https://arxiv.org/abs/2205.06386


∎

11institutetext: A. Badran 22institutetext: Department of Computer Science
Tarbiat Modares University
22email<EMAIL_ADDRESS>33institutetext: Corresponding Author: M.
Rezghi 44institutetext: Department of Computer Science
Tarbiat Modares University
44email<EMAIL_ADDRESS>
# An adaptive synchronization approach for weights of deep reinforcement
learning

S. Amirreza Badran  Mansoor Rezghi

(Received: date / Accepted: date)

###### Abstract

Deep Q-Networks (DQN) is one of the most well-known methods of deep
reinforcement learning, which uses deep learning to approximate the action-
value function. Solving numerous Deep reinforcement learning challenges such
as moving targets problem and the correlation between samples are the main
advantages of this model. Although there have been various extensions of DQN
in recent years, they all use a similar method to DQN to overcome the problem
of moving targets. Despite the advantages mentioned, synchronizing the network
weight in a fixed step size, independent of the agent’s behavior, may in some
cases cause the loss of some properly learned networks. These lost networks
may lead to states with more rewards, hence better samples stored in the
replay memory for future training. In this paper, we address this problem from
the DQN family and provide an adaptive approach for the synchronization of the
neural weights used in DQN. In this method, the synchronization of weights is
done based on the recent behavior of the agent, which is measured by a
criterion at the end of the intervals. To test this method, we adjusted the
DQN and rainbow methods with the proposed adaptive synchronization method. We
compared these adjusted methods with their standard form on well-known games,
which results confirm the quality of our synchronization methods.

###### Keywords:

Reinforcement Learning Deep Learning

##  1 Introduction

Reinforcement learning is a branch of machine learning that deals with how an
agent can act in an environment to receive the maximum rewards from the
environment. One of the well known and first successful neural networks used
in reinforcement learning is Tesauro (1994) ’s TD-gammon (Tesauro, 1994). It
used neural networks to train the agent who won the world backgammon. After
that, one of the most significant breakthroughs in RL was the Deep Q
networkMnih et al. (2015), which used deep learning in the modeling of the
action function. They used convolutional neural networks in combination with
experience replay so that the␣ agent could learn from raw pixels of Atari
games without any handcrafted features nor any changes for a specific game.
Their experiments have shown that the DQN agent could play Atari 2600 as good
as a human.

In addition to using the deep neural networks, addressing the moving targets
problem by enabling the network to be synced after a while rather than
immediate updates, was another success factor of this method. Their approach
for synchronization increased the stability and possibility of convergence to
the optimal policy compared to immediately updating the target network.

Following the success of DQN, many extensions of this method presented in
recent years. For example, double DQN Van Hasselt et al. (2016) addresses the
overestimation bias of Q-learning by separating of selection and evaluation of
actions Hasselt (2010). Prioritized experience replay proposed in Schaul et
al. (2015) improved experience replay of DQN. This improvement is done based
on selecting important experiences more frequently.

The dueling network architecture Wang et al. (2016) is another extension that
tries to generalize across actions by separately representing state values and
action advantages to different neural networks. In distributional Q-learning
Bellemare et al. (2017), the agent learns to approximate the distribution of
returns instead of expected returns.

Recently in Hessel et al. (2018), all of these extensions were integrated into
one agent called Rainbow. In the reported experiments, this integrated method
achieves the state of the art performance.

In all methods similar to the DQN, syncing the deep networks is done at fixed
predefined intervals. Although this approach could solve the moving target
problem, losing some semi-optimal trained target networks in some cases is its
main drawback. Syncing the target network weights without considering the
agent’s behavior may prevent the agent from following a good policy which
could lead to more potential rewards. In this paper, we propose an adaptive
way of syncing with considering the behavior of the agent.

##  2 Background

Reinforcement learning problems can be formalized by the Markov Decision
Process (MDP). MDP is a tuple
$\langle\mathcal{S},\mathcal{A},T,r,\gamma\rangle$ where $\mathcal{S}$ is a
finite set of states, $\mathcal{A}$ is a finite set of actions, $T$ is the
transition matrix, $r$ is the reward function and $\gamma$ is a discount
factor. Usually, in reinforcement learning, an agent starts at state $S_{1}$
and then by selecting an action $A_{t}$, the environment sends a reward signal
$R_{1}$ and the agent moves to the next state $S_{2}$ based on
$T(S_{1},A_{t})$ and so on, until it reaches to a terminal state or the
maximum number of steps allowed.

###  2.1 Basics of reinforcement learning

Policy, value functions, and the model are relevant concepts in RL.

While the policy is the agent’s behavior function, the value functions
determine the quality of each state or each action. The model is the agent’s
representation of the environment. Temporal-Difference learning Sutton (1988)
is One of the most important and widely used techniques in reinforcement
learning. In this method, the agent learns to predict the expected value of
the value function at the end of each step. Expressed formally:

$V(S_{t})\leftarrow V(S_{t})+\alpha[R_{t+1}+\gamma V(S_{t+1})-V(S_{t})].$ (1)

On-policy and off-policy learning methods are two categories that used in RL.
In on-policy learning the value functions us learned from the actions, the
agent achieves using the current policy. In off-policy methods, the policy
used to generate behavior may be unrelated to the policy that is evaluated and
improved.

DQN (based on Q-learnin)g Watkins and Dayan (1992) is an off-policy control
algorithm of TD(0). In this algorithm, first action-values are initialized
arbitrary (e. g. 0) and then in each step old value $Q(s_{t},a_{t})$ is
updated by a fraction (learning rate) of
$r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})$ which called is the
td-target $\delta$ (2). Q-learning is formulated as:

$Q(S_{t},A_{t})\leftarrow Q(S_{t},A_{t})+\alpha\cdot\delta.$ (2)

where $\delta$ is td-target

$\delta=[R_{t+1}+\gamma\max_{a}Q(S_{t+1},a)-Q(S_{t},A_{t})]$ (3)

and $\alpha$ is the learning rate.

Q-learning is considered a tabular solution method Sutton and Barto (2018).
There are other types of RL methods that use approximation instead of a
tabular solution. In Q-learning, we can approximate action-value function $Q$
instead of calculating the exact value of it. This is the main idea of deep
reinforcement learning discussed in section 2.2.

###  2.2 Deep reinforcement learning

The authors inMnih et al. (2015) developed a deep reinforcement learning
method by using a deep convolutional neural network to approximate the action-
value function. Using CNN architecture enabled them to learn from raw images
(in this instance, images of Atari 2600 games) without any handcrafted
features. Furthermore, the same settings and network architecture used from
variant games and DQN showed the state of the art performance in the majority
of the games. The development of DQN was challenging, one of which was moving
targets problem. In each iteration, the values of Q may change, thus altering
the value of the targets, which may cause oscillations or divergence. In Mnih
et al. (2015), to overcome this challenge the weights of the action value
network for C updates are cloned and used as the Q-learning updates. The
duplicated Q-network is showed by $\hat{Q}$. Another challenge of using DQN
was the correlation between successive samples, which led to a large variation
between updates, thus instability of the method. All of mentioned extensions
(including Rainbow) are using the same approach as DQN to overcome the moving
targets problem. Mnih et al. (2015) Modified the standard Q-learning to use
experience replay Lin (1992). Experiences $e_{t}=(s_{t},a_{t},r_{t},s_{t+1})$
at each time-step $t$ are stored in a memory data set
$D_{t}=\\{e_{1},\cdots,e_{t}\\}$ and during learning Q-learning updates will
be applied by samples of these experiences which are selected randomly with a
uniform distribution.

The raw images of Atari games are preprocessed by function $\phi$ (described
in Mnih et al. (2015)) and turned into an $84\times 84\times 4$ image (or
tensor). Q-network consists of three convolutional layers and two fully
connected layers with rectifier nonlinearity (that is, $\max(0,x)$) as
activation function and the output layer has a single output for each valid
action. The Q-network can be trained by optimizing the parameters $\theta_{i}$
at iteration $i$ to reduce the mean-squared error in the Bellman equation,
where the optimal target values
$r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})$ are substituted with
approximate target values
$y=r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime};\theta_{i}^{-})$,
using parameters $\theta_{i}$ from some previous iteration. This leads to a
sequence of loss functions $L_{i}(\theta_{i})$ that changes at each iteration
i,

$L_{i}(\theta_{i})=\mathbb{E}_{s,a,r,s^{\prime}\sim
D_{i}}\bigg{[}(\dot{\delta}-Q(s,a;\theta_{i}))^{2}\bigg{]}$ (4)

where

$\dot{\delta}=r+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime};\theta_{i}^{-})$
(5)

Which can be optimized by gradient descent methods, specifically stochastic
gradient descent method.

Algorithm 1 deep Q-learning with experience replay.

1:Initialize replay memory $D$ to capacity $N$

2:Initialize action-value function $Q$ with random weights $\theta$

3:Initialize target action-value function $\hat{Q}$ with weights
$\theta^{-}=\theta$

4:for $episode=1,\cdots,M$ do

5: Initialize sequence $s_{1}=\\{x_{1}\\}$ and preprocessed sequence
$\phi_{1}=\phi(s_{1})$

6: for $t=1,\cdots,T$ do

7: With probability $\epsilon$ select a random action $a_{t}$

8: otherwise select $a_{t}=argmax_{a}Q(\phi(s_{t}),a;\theta)$

9: Execute action $a_{t}$ in emulator and observe reward $r_{t}$ and image
$x_{t+1}$

10: set $s_{t+1}=(s_{t},a_{t},x_{t+1})$ and preprocess
$\phi_{t+1}=\phi(s_{t+1})$

11: Store transition $(\phi_{t},a_{t},r_{t},\phi_{t+1})$ in $D$

12: Sample random minibatch of transitions $(\phi_{j},a_{j},r_{j},\phi_{j+1})$
from $D$

13: $\textnormal{Set
}y_{j}=\Bigg{\\{}\begin{array}[]{lc}r_{j},&s_{j+1}\textnormal{is terminal }\\\
\dot{\delta},&\textnormal{o.w.}\end{array}$ where
$\dot{\delta}=r_{j}+\gamma\max_{a^{\prime}}\hat{Q}(\phi_{j+1},a^{\prime};\theta^{-})$

14: Perform a gradient descent step on $(y_{j}-Q(\phi_{j},a_{j};\theta))^{2}$
with respect to $\theta$

15: Every $C$ steps reset $\hat{Q}=Q$

##  3 Proposed adaptive synchronization method

As mentioned, in DQN and it’s extensions, syncing the deep networks is done at
intervals with the fixed step size $C$ to solve the moving target problem.
Syncing the target network weights in a fixed step size $C$ without
considering the agent’s behavior may prevent the agent from following a good
policy and leading to more potential rewards. In this section, we define a
criterion to find out when to sync the network instead of the fixed steps
size. This criterion should set up based on the obtained rewards by the agent.
Since these rewards are obtained by the actions of the agent(chosen by
$a_{t}=argmax_{a}\hat{Q}(s_{t},a;\theta)$) then they are dependent to the
Q-network that is learning continuously by target network (see (4)).
Therefore, we can use a stream of recent rewards to measure the behavior of
the agent as an evaluation of the network quality. Then, based on this
evaluation criterion, we can decide when to synchronize the target network
with the Q-network.

Let $t$ be a step that we should decide about syncing. To construct our
criterion for syncing, we push the recent rewards (obtained from the
environment) in a queue $\\{r_{2n-t+1},\ldots,r_{t}\\}$ with the length $2n$
and compare the quality of rewards of the first part, i.e,
$que_{old}=\\{r_{t-2n+1},\cdots,r_{t-n}\\}$ and the second part
$que_{new}=\\{r_{t-n+1},\cdots,r_{t}\\}$ of this queue to construct our
criterion. Here $n$ could be selected as $C$. If the second part works better
than the first one this means that the agent and so the network works good and
we should preserve this network.

Average of rewards can tell us about the behavior of agent but since we are
more interested in recent behavior, applying a weight vector $w$ which
increases the effect of recent rewards can lead to a better representation of
recent behavior of the agent. So this weight vector should satisfy the
following conditions

$\displaystyle w_{1}\leq w_{2}\leq\ldots\leq w_{n},\quad\sum_{i=1}^{n}w_{i}=1$
(6)

Although different type of weights could be used, here we use a simple form

$w_{i}=\frac{i}{n(n+1)}.$

So the weighted average of rewards in both $que_{new}$ and $que_{old}$ will be

$\displaystyle\overline{que}_{new}$
$\displaystyle={\frac{1}{n}}{\sum\limits_{i=1}^{n}w_{i}(que_{new})_{i}}$ (7)
$\displaystyle\overline{que}_{old}$
$\displaystyle={\frac{1}{n}}{\sum\limits_{i=1}^{n}w_{i}(que_{old})_{i}}$ (8)

Now we need to decide whether to sync $\hat{Q}=Q$ or not, for this matter, we
used the simple idea of comparing the weighted average of $que_{new}$ and
$que_{old}$. We, at each $C$ steps, calculate $\overline{que}_{new}$ and
$\overline{que}_{old}$ ((7) , (8)) and if
$\overline{que}_{new}<\overline{que}_{old}$ which means the recent behavior of
the agent is not as good as last time, we sync the weights of Q-network.
Because of adaptability of our synchronization method, we call our modified
DQN agent the Adaptive Synchronization DQN or for short AS_DQN and our
modified Rainbow agent, AS_Rainbow.

Algorithm 2 Adaptive Synchronization DQN

1:Initialize replay memory $D$ to capacity $N$

2:Initialize action-value function $Q$ with random weights $\theta$

3:Initialize target action-value function $\hat{Q}$ with weights
$\theta^{-}=\theta$

4:for $episode=1,\cdots,M$ do

5: Initialize sequence $s_{1}=\\{x_{1}\\}$ and preprocessed sequence
$\phi_{1}=\phi(s_{1})$

6: Initialize $que$ a $2n$ queue containing the rewards with weight vector
$w$.

7: for $t=1,\cdots,T$ do

8: With probability $\epsilon$ select a random action $a_{t}$

9: otherwise select $a_{t}=argmax_{a}Q(\phi(s_{t}),a;\theta)$

10: Execute action $a_{t}$ in emulator and observe reward $r_{t}$ and image
$x_{t+1}$

11: set $s_{t+1}=(s_{t},a_{t},x_{t+1})$ and preprocess
$\phi_{t+1}=\phi(s_{t+1})$

12: Store transition $(\phi_{t},a_{t},r_{t},\phi_{t+1})$ in $D$ and push
$r_{t}$ into $que$

13: Sample random minibatch of transitions $(\phi_{j},a_{j},r_{j},\phi_{j+1})$
from $D$

14: $\textnormal{Set
}y_{j}=\Bigg{\\{}\begin{array}[]{lc}r_{j},&s_{j+1}\textnormal{is terminal }\\\
\dot{\delta},&\textnormal{o.w.}\end{array}$ where
$\dot{\delta}=r_{j}+\gamma\max_{a^{\prime}}\hat{Q}(\phi_{j+1},a^{\prime};\theta^{-})$

15: Perform a gradient descent step on $(y_{j}-Q(\phi_{j},a_{j};\theta))^{2}$
with respect to $\theta$

16: if $mod(step,C)==0$ then

17:
$\overline{que}_{new}={\frac{1}{n}}{\sum\limits_{i=1}^{n}w_{i}(que_{new})_{i}}$

18:
$\overline{que}_{old}={\frac{1}{n}}{\sum\limits_{i=1}^{n}w_{i}(que_{old})_{i}}$

19: if $\overline{que}_{new}<\overline{que}_{old}$ then

20: Reset $\hat{Q}=Q$

##  4 Experimental results

In this section, we describe our settings and configurations used to test our
agents. We used a Linux-based PC to train our agents, and each session lasts
between 3 and 4 days. For DQN implementation, we used the Dopamine Castro et
al. (2018) library and modified their DQN and Rainbow implementations with our
proposed method. For the testing environment, we used Atari 2600 games
provided by OpenAi’s Gym Brockman et al. (2016).

###  4.1 Evaluation methodology

We did the experiments on AirRaid, Alien, Amidar, Assault, Asterix, Asteroids,
Breakout, SpaceInvaders, and Seaquest games. We tried to choose games that are
different in visuals and gameplay. In our all experiments all parameters (even
the target network period $C$ ) were same as DQN Mnih et al. (2015) and
Rainbow Hessel et al. (2018) accordingly and for our hyper-parameters we
tested our agent with $k=3$ and $\delta=0.01$ .Also, similar to Mnih et al.
(2015) and Hessel et al. (2018) we trained our agents with 200 iterations,
where each iteration consists of two training and evaluation phases. In the
training phase, the agent is trained on 1M frames, and in the evaluation
phase, the agent is evaluated only on 512k frames. In both phases, we used the
mean of episode return (scores) to evaluate our agents and compare them to DQN
and Rainbow. For DQN and Rainbow results, we used data prepared by Dopamine
cite r51. Also, the reported results for each game are an average of 5
training sessions. Although due to limited computing tools and time, we only
trained our agent for 3 sessions in 4 games, in all 9 games, our agents were
even better than the best DQN representatives.

###  4.2 Analysis

First, we trained our agent on the Breakout game for 50 iterations. Then we
plot the real-time return for 1 million frames (or 250k steps) and marked when
our agent synced its target network (figure 1). The agent behavior was as
expected. It maintained the target networks which it achieved higher returns
and synced to new ones when the performance did not meet the criteria.

Figure 1: Real-time return of AS_DQN agent after 50 iterations of training.
Red marks mean the agent did not sync its target network and green marks mean
it did.

As mentioned before, we tested our agents on 9 Atari 2600 games. We used DQN
baselines provided by Dopamine library Castro et al. (2018). AS_DQN
outperformed DQN in almost all 9 games. To be more specific, the average
rewards of 3 AS_DQN agents outperformed the average of 5 DQN agent on 4 games
(figure 2). Also, AS_DQN agents had less variance over different runs (e.g.
the variance of DQN returns over 5 runs of Seaquest is 916200 compared to
AS_DQN’s 590507). Furthermore, our single AS_DQN agents outperformed the
average of 5 DQN agents on 5 other games (figure 3).

AS_Rainbow shows an exceptionally good performance in Breakout and Seaquest
(Figures 4(b) and 4(d)) but Rainbow outperformed AS_Rainbow in Asterix (Figure
4(a)) and AS_Rainbow performs better than Rainbow in SpaceInvaders (Figure
4(c)).

(a) Asterix

(b) Breakout

(c) SpaceInvaders

(d) Seaquest

Figure 2:  DQN vs AS_DQN. Average return of episodes for each iteration are
shown for 4 different games. DQN results are averaged over 5 runs and AS_DQN
runs are averaged over 3 runs in this figure.

(a) AirRaid

(b) Alien

(c) Amidar

(d) Assault

(e) Asteroids

Figure 3:  DQN vs AS_DQN. Average return of episodes for each iteration are
shown for 5 different games. DQN results are averaged over 5 runs and AS_DQN
runs are averaged over only one in this figure.

(a) Asterix

(b) Breakout

(c) SpaceInvaders

(d) Seaquest

Figure 4:  Rainbow vs AS_Rainbow. Average return of episodes for each
iteration are shown for 4 different games. DQN results are also shown for
better insight of Rainbow and AS_Rainbow performances.

(a) Asterix

(b) Breakout

Figure 5:  DQN vs AS_DQN. Average return of episodes for each iteration are
shown for 2 different games with $C=100,1000,3000,4000$ and $C=8000$ as
benchmark.

The main problem in methods that use the target network is finding the optimal
length of fixed intervals. If we choose a very small step size $C$ for fixed
intervals, it leads to instability during training the Q-network, which is not
desirable(see DQN agent with $C=100$ in figure 5); And if we choose a very
long step size $C$, it may lead to overfitting the Q-network to every target
network, which again, is not desirable. On the other hand, scale of these step
sizes is different for every problem and environment, as you may already know,
Atari’s Pong game can be trained by a small step size (e.g. $C=100$) which, if
used for some other game (e.g. SpaceInvaders) leads to divergence from optimal
policy. Therefor, finding appropriate step size is an important problem for
DQN and methods based on it and considering only a predefined step size is not
optimal. Despite this, due to our knowledge all state of the art methods like
Rainbow based on DQN are using a fixed step size. To overcome this problem the
main question is that how we define a criterion for quality of a target
network? Is the instant rewards that agent obtains from environment a good
criterion? These rewards are obtained by actions of the agent, which it
chooses based on the current Q-network that is trained by target network. If
we use a moving average over recent rewards we can eliminate any instant
reward noises and have an acceptable (based on results) criterion.

At last we desire to sync the target network as soon as possible, but in fixed
step sized method, networks sync at a predetermined rate. In our method if we
choose smaller step sizes, the agent tend to sync the networks as soon as the
criteria allows it, and thats why our agent performs better at step sizes
lower than $C=8000$ (figure 5).

##  5 Conclusion

In this paper, we investigated the DQN Reinforcement learning methods and
considered its approach to solve moving targets problem. Here we discussed the
drawback of syncing the network in fixed step size used by DQN and its
extension, then shown that by evaluating the behavior of agent one could
propose an adaptive syncing method. Optimally we want to sync the target
network as soon as possible, but by reducing the step size $C$ we risk the
stability of the agent learning process. This type of adaptive approach
discussed in this paper, might be the answer to this problem, no matter what
is our environment (Atari or not), Our agents could have stable learning
process towards the optimal policy without finding and fine tuning a fixed
step size. For this matter, we proposed a simple method for this evaluation
and experiments show the quality of this simple evaluation criteria. It is
clear that in the future, we and others could derive more sophisticated
criterion for the quality of target networks. In our test we did not change
any hyper-parameters which were used in Mnih et al. (2015) and Hessel et al.
(2018) to make a fair comparison. Due to time and hardware limitations our
experiment results are not many but still enough to prove our method is
effective.

## References

  * Bellemare et al. (2017) Bellemare MG, Dabney W, Munos R (2017) A distributional perspective on reinforcement learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp 449–458
  * Brockman et al. (2016) Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, Zaremba W (2016) Openai gym. arXiv preprint arXiv:160601540
  * Castro et al. (2018) Castro PS, Moitra S, Gelada C, Kumar S, Bellemare MG (2018) Dopamine: A Research Framework for Deep Reinforcement Learning URL http://arxiv.org/abs/1812.06110
  * Hasselt (2010) Hasselt HV (2010) Double q-learning. In: Advances in Neural Information Processing Systems, pp 2613–2621
  * Hessel et al. (2018) Hessel M, Modayil J, Van Hasselt H, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2018) Rainbow: Combining improvements in deep reinforcement learning. In: Thirty-Second AAAI Conference on Artificial Intelligence
  * Lin (1992) Lin LJ (1992) Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning 8(3-4):293–321
  * Mnih et al. (2015) Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, et al. (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529
  * Schaul et al. (2015) Schaul T, Quan J, Antonoglou I, Silver D (2015) Prioritized experience replay. arXiv preprint arXiv:151105952
  * Sutton (1988) Sutton RS (1988) Learning to predict by the methods of temporal differences. Machine learning 3(1):9–44
  * Sutton and Barto (2018) Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press
  * Tesauro (1994) Tesauro G (1994) Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation 6(2):215–219
  * Van Hasselt et al. (2016) Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Thirtieth AAAI Conference on Artificial Intelligence
  * Wang et al. (2016) Wang Z, Schaul T, Hessel M, Van Hasselt H, Lanctot M, De Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, JMLR.org, ICML’16, pp 1995–2003, URL http://dl.acm.org/citation.cfm?id=3045390.3045601
  * Watkins and Dayan (1992) Watkins CJ, Dayan P (1992) Q-learning. Machine learning 8(3-4):279–292