aaaffiliationtext: Department of Mathematics, Illinois State University,
USAbbaffiliationtext: School of Mathematics and Statistics, Jiangsu Normal
University, China

The reporting delay in data breach incidents poses a formidable challenge for
Incurred But Not Reported (IBNR) studies, complicating reserve estimation for
actuarial professionals. This work presents a novel Bayesian nowcasting model
designed to accurately model and predict the number of IBNR data breach
incidents. Leveraging a Bayesian modeling framework, the model integrates time
and heterogeneous effects to enhance predictive accuracy. Synthetic and
empirical studies demonstrate the superior performance of the proposed model,
highlighting its efficacy in addressing the complexities of IBNR estimation.
Furthermore, we examine reserve estimation for IBNR incidents using the
proposed model, shedding light on its implications for actuarial practice.

##  1 Introduction

Data breaches pose a significant threat to computer systems and have become an
ongoing problem due to the extensive network activities. Numerous severe
cybersecurity incidents have demonstrated the gravity of this issue. For
instance, the Privacy Rights Clearinghouse (PRC) has reported over 11 billion
records breached since 2005. In 2022, the Identity Theft Resource Center
(ITRC) recorded 1,802 data breach incidents, affecting approximately
422,143,312 individuals. Similarly, in 2021, there were 1,862 incidents
reported by the ITRC, impacting around 293,927,708
individuals***https://www.idtheftcenter.org/publication/2022-data-breach-
report/. The cost associated with data breaches is also considerable.
According to the Cost of a Data Breach Report 2023 published by
IBM†††https://www.ibm.com/reports/data-breach ‡‡‡The Cost of a Data Breach
Report 2023, conducted independently by Ponemon Institute and sponsored,
analyzed, and published by IBM Security, examined 553 organizations affected
by data breaches occurring between March 2022 and March 2023., the average
cost of a data breach incident increased from $4.35 million in 2022 to $4.45
million in 2023. Additionally, the average cost per breached record saw a 0.6%
increase from 2022 to 2023, reaching $165 compared to $164. Taking a long-term
perspective, the average total cost of a data breach has risen by 15.3% since
2020, when it was $3.86 million.

The unique nature of cyber risk often leads to the discovery of data breaches
days, months, or even years after the initial breach occurred. In the 2023 IBM
report, it is stated that the mean time to identify a data breach from 2021 to
2023 was 212, 207, and 204 days, respectively. Similarly, the mean time to
contain a data breach was 287, 277, and 277 days, respectively. As breaches go
unaddressed for longer periods, more data is leaked, resulting in a larger
overall impact, both financially and otherwise. For instance, in 2023, a
breach with a lifecycle exceeding 200 days had an average cost of $4.95
million, whereas breaches with a lifecycle of less than 200 days cost an
average of $3.93 million. This poses a significant challenge for estimating
reserves for incurred but not reported (IBNR) breach incidents. Solvency II
regulations require reserves to be set based on the best estimate of the loss
distribution, which further complicates the IBNR study. It should be noted
that there are many studies on the IBNR in the actuarial and insurance
literature [3, 22, 21, 7]. However, there is limited research on IBNR studies
for cyber data breach incidents in the existing literature. One of the most
relevant works in this area is by Xu and Nguyen [23], where they developed a
novel statistical approach to model the time to identify an incident and the
time to notice affected individuals. However, their study did not specifically
address the IBNR issue, which is the focus of the current study.

Other studies offer tangential insights into data breach dynamics. For
example, Buckman et al. [5] studied the time intervals between data breaches
for enterprises that had at least two incidents between 2010 and 2016. They
showed that the duration between two data breaches might increase or decrease,
depending on some factors. Edwards et al. [8] analyzed the temporal trend of
data breach size and frequency and showed that the breach size follows a log-
normal distribution and the frequency follows a negative binomial
distribution. They further showed that the frequency of large breaches (over
500,000 breached records) follows the Poisson distribution, rather than the
negative binomial distribution, and that the size of large breaches still
follows a log-normal distribution. Eling and Loperfido [9] studied data
breaches from actuarial modeling and pricing perspectives. They used
multidimensional scaling and goodness-of-fit tests to analyze the distribution
of data breaches. They showed that different types of data breaches should be
analyzed separately and that breach sizes can be modeled by the skew-normal
distribution. Sun et al. [19] developed a frequency-severity actuarial model
of aggregated enterprise-level breach data to promote ratemaking and
underwriting in insurance, and they also developed a multivariate frequency-
severity model to examine the status of data breaches at the state level in
[20]. Ikegami and Kikuchi [12] studied a breach dataset in Japan and developed
a probabilistic model for estimating the data breach risk. They showed that
the inter-arrival times of data breaches (for those enterprises with multiple
breaches) follow a negative binomial distribution. Romanosky et al. [18] used
a fixed effect model to estimate the impact of data breach disclosure policy
on the frequency of identity thefts incurred by data breaches. The proposed
study is significantly different from those in the literature since we aim to
address the important yet unstudied problem–IBNR cyber incidents.

Nowcasting, or predicting the present, is an effective method for estimating
the IBNR events in real-time, using an incomplete set of reports. This
approach has been widely utilized in various fields, including disease
surveillance and weather forecasting [16, 2]. By leveraging available data and
statistical techniques, nowcasting enables analysts to estimate the current
state of events that may not have been fully reported or observed yet. This
methodology can be particularly useful in estimating the number of IBNR events
in the context of data breaches, where the reporting of incidents may be
delayed or incomplete. In this study, we develop a novel Bayesian nowcasting
approach to modeling and predicting the number of IBNR data breach incidents
within one year. By examining temporal and heterogeneous effects on IBNR
modeling, this research aims to enhance the accuracy of reserve estimations
for cyber incidents.

The rest of the paper is organized as follows. Section 2 introduces the data,
and performs the exploratory data analysis. In Section 3, we introduce the
proposed Bayesian nowcast model. Section 4 examines the model performance on
synthetic data. In Section 5, we employ the proposed model to the empirical
data and present the reserve estimation. The last Section concludes the
current work and present some discussion.

##  2 Data Preprocessing and Exploratory Data Analysis

The incident data was collected from the ITRC, which gathers information about
publicly reported data breaches from various sources, including company
announcements, mainstream news media, government agencies, recognized security
research firms and researchers, and non-profit organizations. We focus on
recent incidents with breach dates from 1/1/2018 to 12/31/2023, and drop the
incidents without the information of affected individuals. After further
removing the duplicates, we have a total of 859 incidents.

Figure 1 shows the data structure for the reporting delay of cyber incidents
based on the breach data. The rows correspond to time $t=1,2,\ldots,T$, and
$T$ is the present time. The columns correspond to the amount of delay in the
same unit as $t$. In our data, the unit is set to be one month. $n_{t,D}$
represents the number of incidents was reported between $(D-1)$th and $D$th
months after occurrences at time $t$. For example, for the first row,
$n_{1,1}$ represents the number of breach incidents that occurred during the
first month of our study period and were reported within the month after they
occurred, $n_{1,2}$ incidents were reported between the second month after
occurrence, and $n_{1,3}$ incidents were reported between the second and third
month after occurrence. In our study, we chose $D=12$ because one year is a
typical choice for the reserving period [22]. That is, the incidents with a
delay larger than one year were excluded from the current study. The last
column $n_{t}$ represents the total incidents occurring at time period $t$.
Since the current time is $T$, the completely observable times are from 1 to
$T-D$. From $T-D+1$ to $T-1$, we can only observe partial observations and
need to nowcast the incidents (i.e., determining the number of IBNR
incidents).

Figure 1: Data structure in the reporting delay of cyber incidents. The values
in the gray area represent observed data, and the values in the gray area
represent the number of incurred but not reported (IBNR) events. Unit: month.

The dataset studied is divided into 72-month time periods. For the purpose of
assessing model performance, we use the first 60 months as the in-sample data
(i.e., 1/1/2018 to 12/31/2022), and the rest of the data as the out-of-sample
data (i.e. 1/1/2023 to 12/31/2023), i.e., $T=61$.

(a) Time series

(b) Boxplots

Figure 2: Time series plots and boxplots of frequencies of the number of data
breaches reported with different delay periods. The red dot in the boxplot
represents the mean.

Figure 2(a) illustrates the time series plots of the frequency of the number
of data breaches reported with different delay periods for the in-sample data.
The number of incidents in different delayed months changes over time, showing
an increasing trend in recent months. Particularly, we see a jump in December
2022, (i.e., 155 incidents occurred and were reported in this month). Most of
the incidents were delayed in the first three months. Figure 2(b) further
shows the boxplots of frequency in various delayed months. It is clearly seen
that the frequency shows a decreasing trend over delayed months. This suggests
that the time effect needs to be incorporated into the modeling process.

Figure 3: Correlation plots of frequencies in different delayed months.

Figure 3 shows the correlation plots of the frequency among different delayed
months. We observe correlations among adjacent months. It is interesting to
see the high correlations among large adjacent delayed months (e.g., 10/11/12
months), which is because there are a few incidents reported in those large
delayed months. This suggests that the correlations among different delayed
months should be considered as well.

##  3 Bayesian Nowcasting Model

The Bayesian nowcasting model proposed in this study aims to capture the time
effect and correlations among the delayed months in the number of data breach
events that occur but are not yet reported. Define a random variable $n_{t,d}$
that represents the number of events occurring at time $t$ (where
$t=1,2,\ldots,T-1$) but reported until time $d$ (where $d=1,2,\ldots,D$). The
model assumes that $n_{t,d}$ follows a negative binomial distribution:

$n_{t,d}\sim\text{NB}(p_{t,d},r_{t,d}),$ (1)

where $p_{t,d}$ is the probability of breach and $r_{t,d}$ is the number of
breaches. The proposed distribution is very flexible as it allows the
probability and number of branches to vary with the time and the delayed
month. To capture the effect and correlations among the delayed months, we
propose the following Bayesian model

${\rm logit}(p_{t,d})=\alpha_{0}+\alpha_{1}t+\alpha_{2}d/D,$ (2)

and

$\log(r_{t,d})=\beta_{0}+\beta_{1}t+\beta_{2}d/D,$ (3)

where

$\alpha_{i}\sim N\left(0,\sigma^{2}_{\alpha_{i}}\right),\quad i=0,1,2,$

and

$\beta_{j}\sim N\left(0,\sigma^{2}_{\beta_{j}}\right),\quad j=0,1,2.$

Eq. (2) models the breach probability with logistic regression with random
effect effects, and Eq. (3) models the log-transformed number of incidents
with a regression model with random effects. The prior distributions for the
random effects are assumed to be normal distributions. The variability of the
random effects is incorporated by assigning prior distributions, a standard
exponential distribution, to the parameters $\sigma_{\alpha_{i}}$ and
$\sigma_{\beta_{j}}$. That is,

$\sigma_{\alpha_{i}}\sim{\rm Exp}(1),\quad\sigma_{\beta_{j}}\sim{\rm Exp}(1),$

for $i,j=0,1,2$. The posterior distribution for the model parameters
$\Theta=(\alpha_{i},\beta_{j},\sigma_{\alpha_{i}},\sigma_{\beta_{j}})$ can be
represented as:

$p(\Theta|\mathbf{n})=p(\Theta)\prod_{t=1}^{T-1}\prod_{d=1}^{D}p(n_{t,d}|\Theta),$
(4)

where $p(\Theta|\mathbf{n})$ follows the negative binomial distribution and
$\mathbf{n}$ represents all the observed data. The joint prior distributions
$p(\Theta)$ are specified for the model parameters. The estimation of
posterior distributions is performed using Markov Chain Monte Carlo (MCMC)
methods [13, 6] .

In our study, the IBNR nowcasting is done by estimating the posterior
predictive distribution:

$p(n_{t,d})=\int_{\Theta}p(n_{t,d}|\Theta)p(\Theta|\mathbf{n})d\Theta,$ (5)

where the MCMC approach is employed to obtain samples from the posterior
predictive distribution.

##  4 Sythetic data study

To understand the model performance, we examine the model performance on a
synthetic dataset. The data is generated from a Negative binomial distribution
with

${\rm logit}(p_{t,d})=-1.5-.01t+.8d/D,$ $\log(r_{t,d})=1.5+.01t-1.8d/D,$

where $t=1,\ldots,72$, and $d=1,\ldots,12$, $D=12$.

(a) Time series

(b) Boxplots

Figure 4: Time series plots and boxplots of frequencies of the number of data
breaches reported with different delay periods. The red dot in the boxplot
represents the mean.

The time series plots of delayed months are displayed in Figure 4(a). It is
observed that a significant portion of simulated incidents is reported within
the initial months following the occurrence. The time series shows variability
and an increasing trend in recent months. The boxplots in Figure 4(b) reveal
that the data is skewed, as indicated by the comparatively large means.
Furthermore, there is large variability across the dataset, particularly
within the initial months of delay (up to 4 delayed months). Additionally, the
presence of outliers within the simulated sample further underscores the
heterogeneity of incident reporting patterns.

(a) $\alpha_{0}$

(b) $\alpha_{1}$

(c) $\alpha_{2}$

Figure 5: Trace plots of parameters in Eq. (2).

(a) $\beta_{0}$

(b) $\beta_{1}$

(c) $\beta_{2}$

Figure 6: Trace plots of parameters in Eq. (3).

We use the following setting in the model fitting. Three MCMC chains are used
to estimate the posterior distributions to ensure convergence and assess the
robustness of the results. The initial values are randomly generated based on
prior distributions. The burn-in sample is set to be $1e+06$, while the number
of iterations is $1.1e+06$. To reduce autocorrelation between samples, we use
100 as the thinning parameter, i.e., every 100th sample will be retained from
each chain.

To evaluate the convergence of the Markov chains under consideration, we
examine the trace plots for the parameters of Eq. (2) presented in Figure 5
and for the parameters of Eq. (3) shown in Figure 6. These plots reveal that
all three chains demonstrate patterns indicative of convergence. In addition
to visual inspection of the trace plots, we quantitatively assess convergence
using the Potential Scale Reduction Factor (PSRF). The PSRF provides a
comparative measure of the variance within each chain to the variance across
chains, as proposed by Brooks and Gelman [11, 4]. For all the parameters under
study, the PSRF values are uniformly below 1.01, with the multivariate PSRF
also recorded at 1.01. These findings provide robust evidence of convergence
across the MCMC chains in our analysis, underscoring the reliability of the
sampling process in capturing the underlying statistical properties of the
model.

Table 1: Posterior means and 95% Bayesian credibility intervals for parameters. Parameters | Mean | 95% Credibility Interval | True
---|---|---|---
$\alpha_{0}$ | -1.5344 | (-1.9026, -1.2040) | -1.5
$\alpha_{1}$ | -0.0094 | (-0.0164, -0.0026) | -0.01
$\alpha_{2}$ | 0.9545 | (0.4804, 1.4330) | 0.8
$\beta_{0}$ | 1.4265 | (1.0636, 1.7550) | 1.5
$\beta_{1}$ | 0.0118 | (0.0053, 0.0184) | 0.01
$\beta_{2}$ | -1.6673 | (-2.1317, -1.2123) | -1.8
$\sigma_{1}$ | 0.7244 | (0.0497, 2.2164) | –
$\sigma_{2}$ | 1.5081 | (0.1085, 4.7381) | –
$\sigma_{3}$ | 0.6274 | (0.0432, 2.0101) | –
$\sigma_{4}$ | 0.7031 | (0.0492, 2.1804) | –
$\sigma_{5}$ | 1.5195 | (0.1151, 4.8460) | –
$\sigma_{6}$ | 1.0358 | (0.0713, 3.2423) | –

Table 1 presents the posterior means and corresponding 95% Bayesian
credibility intervals for the estimated parameters. We observe that the
posterior means demonstrate a close proximity to the true values, indicating a
robust performance of the estimation procedure. Furthermore, all 95% Bayesian
credibility intervals encompass the true parameter values, signifying the
reliability and accuracy of the Bayesian inference process.

Figure 7: Nowcast the IBNR cyber incidents.

An advantage of employing the Bayesian approach is the direct simulation of
nowcasts from the model itself. In our analysis, we utilize the simulated
median as the prediction. Figure 7 illustrates a scatter plot comparing the
nowcasted frequencies with the real frequencies. Upon inspection, it becomes
apparent that there are instances where the model underpredicts the
frequencies. One possible explanation for this discrepancy could be the
presence of outliers in the synthetic data, as observed in Figure 4. Despite
the presence of outliers, the Pearson correlation coefficient of 0.548
suggests a reasonably strong positive correlation between the nowcasted and
real frequencies. This correlation indicates that the proposed model
demonstrates good predictive performance when applied to synthetic data, even
in the presence of outliers.

##  5 Empirical study

In this section, we employ the proposed model to the ITRC data. Recall that
the in-sample data is from 1/1/2018 to 12/31/2022, while the out-of-sample
data is from 1/1/2023 to 12/31/2023.

###  5.1 Model fitting

In our Bayesian model, we employ three chains to estimate posterior
distributions, initializing them with random values drawn from standard
exponential distributions based on prior information. We set the burn-in
sample to $2\times 10^{6}$, with a total number of iterations at $2.1\times
10^{6}$. Again, we utilize a thinning parameter of 100 to ensure efficient
sampling.

Examining the trace plots for the parameters depicted in Figure 8 and Figure
9, it is evident that all three chains exhibit patterns suggestive of
convergence. These trace plots provide a visual representation of how the
parameter values evolve over the course of the sampling process. The
convergence patterns observed in the trace plots indicate that the chains have
reached stable distributions, suggesting that the MCMC algorithm has
successfully explored the posterior space and obtained reliable estimates of
the model parameters.

(a) $\alpha_{0}$

(b) $\alpha_{1}$

(c) $\alpha_{2}$

Figure 8: Trace plots of parameters in Eq. (2).

(a) $\beta_{0}$

(b) $\beta_{1}$

(c) $\beta_{2}$

Figure 9: Trace plots of parameters in Eq. (3).

For all parameters investigated, the PSRF values consistently register below
1.01, with the multivariate PSRF also measured at 1.01. These results further
offer compelling evidence of convergence across the MCMC chains utilized in
our analysis.

Table 2:  Posterior means and 95% Bayesian credibility intervals for parameters. Parameter | Mean | 95% Credibility Interval
---|---|---
$\alpha_{0}$ | -1.1895 | (-1.6363, -0.7578)
$\alpha_{1}$ | -0.0045 | (-0.0139, 0.0050)
$\alpha_{2}$ | 0.7907 | (0.2664, 1.2950)
$\beta_{0}$ | 1.4315 | (0.9691, 1.8858)
$\beta_{1}$ | 0.0129 | (0.0031, 0.0226)
$\beta_{2}$ | -1.7282 | (-2.2510, -1.2183)
$\sigma_{\alpha_{0}}$ | 0.7220 | (0.0515, 2.3004)
$\sigma_{\alpha_{2}}$ | 1.5221 | (0.1031, 4.9042)
$\sigma_{\alpha_{3}}$ | 0.6032 | (0.0472, 1.9400)
$\sigma_{\beta_{0}}$ | 0.8610 | (0.0555, 2.7244)
$\sigma_{\beta_{1}}$ | 1.5246 | (0.1277, 4.6604)
$\sigma_{\beta_{2}}$ | 1.1488 | (0.0743, 3.7174)

Table 2 displays the posterior mean and 95% Bayesian credibility intervals for
the parameters within our proposed model. The parameter estimates $\alpha_{1}$
and $\alpha_{2}$ exhibit varying degrees of influence on the model’s dynamics.
Specifically, $\alpha_{1}$ presents a negative impact, suggesting an increased
difficulty in system breaches over recent years. In contrast, the positive
effect of $\alpha_{2}$ indicates that, when breaches do occur, their discovery
might be delayed, reflecting potentially slower detection processes or more
sophisticated breach tactics. Further analysis of the $\beta$ parameters
reveals additional insights. $\beta_{1}$ shows a positive effect, suggesting
an increase in the frequency of incidents in recent years. Conversely,
$\beta_{2}$ indicates a negative effect, which could be interpreted as a
reduction in the number of incidents with large delays before discovery. While
the mean estimate for $\alpha_{1}$ might initially suggest a minimal impact on
the model, its inclusion enhances the model’s predictive accuracy. This
highlights the nuanced role $\alpha_{1}$ plays in the model, potentially
capturing dynamics in system breach incidents and their detection timelines.
Therefore, despite its seemingly minor direct impact, we include $\alpha_{1}$
in our analysis.

###  5.2 Nowcasting performance

In our model, the median of simulated samples is used as the prediction.

Figure 10: Nowcast the IBNR cyber incidents. The ‘1m’ line represents the
number of cyber incidents that are reported during the occurring month. The
dashed lines of ‘95 lower bound’ and ‘95 upper bound’ represent the 95%
prediction bounds based on the proposed model. The observed number of
incidents represents all the incidents that have been observed until December
2022.

Figure 10 presents the nowcasting outcomes accompanied by 95% prediction
intervals, juxtaposed with the observed incident counts. The ‘1m’ line reveals
a scant number of incidents reported during the respective occurring months.
The observed incidents for any given month reflect the cumulative count of
occurrences discovered until the end of December 2022. For example, in January
2022, 5 incidents were reported, while February 2022 saw 18 reported
incidents, all of which had transpired in January 2022. Thus, the total number
of incidents occurring in January 2022 and reported by the end of December
2022 amounted to 91. Similarly, February 2022 saw 10 reported incidents, yet
the total observed incidents for that month by the end of December 2022 stood
at 75. The nowcast line delineates the total incidents occurring in a specific
month and reported within 12 months. For instance, the total number of
incidents occurring in February 2022 and reported within 12 months, as
estimated by the proposed model, was 77. Conversely, the ‘real’ line portrays
the total incidents occurring in a specific month and observed within 12
months, based on testing data. In the case of February 2022, the total
observed incidents by the end of January 2023 amounted to 77. Figure 10
illustrates a close alignment between the nowcast and real incident counts
during the initial months, with slight deviations in subsequent months. This
discrepancy can be attributed to the reduced number of observations in later
months, which is evident from the wider prediction intervals.

Figure 11: Nowcast the IBNR cyber incidents.

Similarly, we apply the Bayesian approach to nowcast the delayed incidents,
utilizing the simulated median as the prediction. Figure 11 presents a scatter
plot comparing the nowcasted frequencies with the real frequencies.
Remarkably, a high correlation between the two is evident, with the Pearson
correlation coefficient reaching 0.875. This substantial correlation suggests
that the proposed model demonstrates robust predictive performance.

#### Model comparison.

To further assess the model performance, we compare the proposed model
(labeled as M0) with other commonly used ones in the literature.

  * •

M1: Chain-ladder model [1]. This is the most commonly used approach in
practice. It assumes that the proportional developments of claims from one
development period to the next are the same for all origin periods.

  * •

M2: Mack chain-ladder model [14, 15]. It is a distribution-free stochastic
model underlying the Chain-ladder approach. Let $S_{ik}$ denote the cumulative
claims of origin period (e.g. accident year) $i=1,\ldots,m$, with claims known
for development period (e.g. development year) $1\leq k\leq n+1-i$. In order
to forecast the amounts $S_{ik}$ for $n+1-i<k\leq n$, the Mack chain-ladder
model assumes:

$\displaystyle{\rm E}[F_{i,k}|S_{i1},S_{i2},\ldots
S_{ik}]=f_{k}\leavevmode\nobreak\ {\rm with}\leavevmode\nobreak\
F_{i,k}=\frac{S_{i,k+1}}{S_{ik}},$ $\displaystyle{\rm
Var}[F_{i,k}|S_{i1},S_{i2},\ldots
S_{ik}]=\frac{\sigma_{k}^{2}}{w_{ik}S_{ik}^{\alpha}},$
$\displaystyle(S_{i1},S_{i2},\ldots S_{in})\leavevmode\nobreak\ \mbox{are
independent by}\leavevmode\nobreak\ i,\leavevmode\nobreak\ 1\leq i\leq n$

with $w_{ik}\leq[0,1]$, $\alpha\in\\{0,1,2\\}$.

  * •

M3: Bootstrap chain-ladder model [10]. It employs a two-stage bootstrapping
methodology to forecast the trajectory of the development run-off. In the
initial stage, the model applies the ordinary chain-ladder method to the
cumulative claims triangle. Subsequently, scaled Pearson residuals are
computed from this stage, and through bootstrapping iterations ($R=1000$ times
in our study), future incremental claims payments are projected using the
standard chain-ladder method. Moving to the second stage, the process error is
simulated by incorporating the bootstrapped values as means, while adhering to
the assumed process distribution. The resulting set of reserves derived from
this approach constitutes the predictive distribution. For the bootstrap
chain-ladder model, we use the median as the prediction.

  * •

M4: Generalized linear model. Renshaw and Verrall [17] proposed a methodology
for modeling incremental claims utilizing an "over-dispersed" Poisson
distribution. If we denote the incremental claims for origin year $i$ in
development year $k$ as $C_{ik}$, the model is characterized by the following
equations:

$\mathbb{E}[C_{ik}]=m_{ik},\quad\text{and}\quad\text{Var}[C_{ik}]=\phi\mathbb{E}[C_{ik}]=\phi
m_{ik},$

where $\phi$ is a scale parameter, $\log(m_{ik})=\eta_{ik}$, and
$\eta_{ik}=c+\alpha_{i}+\beta_{k}$, with $\alpha_{1}=\beta_{1}=0$. It is a
generalized linear model wherein the response is modeled using a logarithmic
link function, and the variance is directly proportional to the mean.

Model comparison based on root Mean Squared Error (MSE) and Mean Absolute
Error (MAE) is a common approach to assessing the predictive performance of
different models. MSE measures the average squared difference between
predicted values and actual values, while MAE measures the average absolute
difference between predicted values and actual values.

Table 3: Prediction results of IBNR incidents from various models. | M0 | M1 | M2 | M3 | M4
---|---|---|---|---|---
RMSE | 3.0063 | 26.0646 | 7.2199 | 7.4944 | 16.6219
MAE | 2.2727 | 14.4414 | 5.7634 | 5.4091 | 8.4394

Referring to Table 3, which displays the prediction results from various
models, it is evident that the performance of the models varies significantly.
The proposed model M0 exhibits the lowest RMSE and MAE, indicating superior
predictive accuracy compared to the other models. Conversely, Model M1
demonstrates the highest RMSE and MAE values, suggesting poorer predictive
performance. Models M2 and M3 fall within intermediate ranges of RMSE and MAE
values, while Model M4 has a poor prediction performance.

In summary, the proposed model demonstrates satisfactory nowcasting
performance, effectively estimating incident counts with high accuracy.

###  5.3 Reserve Estimation

We apply the results derived from our proposed Bayesian model to estimate the
financial impact of cyber incidents, leveraging the average cost of a data
breach. According to IBM’s 2022 report, the average cost of a data breach
incident is estimated at $4.35 million. This value serves as the per-incident
cost in our analysis. By integrating this cost metric with the model’s now-
cast incident frequency, we can approximate the total financial impact and the
insurance reserves that are expected to cover over the year.

Table 4 presents the reserve estimation for cyber incidents in 2022, employing
the proposed Bayesian model. We calculate monthly figures for estimated, paid,
IBNR, and ultimate claims, all denominated in millions based on the nowcast
frequencies, where the ultimate claims are computed by using the out-of-sample
data of 2023. A consistent trend observed over the year is the widening
discrepancy between estimated and paid claims, leading to a significant
accumulation of IBNR claims. The peak in estimated claims was witnessed in
December, surging to 1150.575 million, alongside a notable increase in IBNR
claims to 476.325 million. It is interesting to see that the IBNR rate shows
an overall decreasing trend. The comparative analysis of estimated versus
ultimate claims underscores the good performance of the proposed Bayesian
model in managing the uncertainties in cyber incident claims. With estimated
total claims reaching 7490.7 million, slightly above the ultimate claims of
7312.35 million, the model exhibits a marginally conservative bias.
Nevertheless, this nuance further underscores the model’s significant promise
in refining actuarial methodologies for cyber risk management.

Table 4: Reserve estimation for cyber incidents in 2022 with IBNR Change Rate. The IBNR(%) represents the IBNR change rate. Unit: million. Month | Estimated Claims | Paid Claims | IBNR Claims | Ultimate Claims | IBNR(%)
---|---|---|---|---|---
01/22 | 395.850 | 395.85 | 0.000 | 395.85 | –
02/22 | 334.950 | 326.25 | 8.700 | 334.95 | –
03/22 | 450.225 | 430.65 | 19.575 | 439.35 | 125.00
04/22 | 500.250 | 465.45 | 34.800 | 526.35 | 77.78
05/22 | 504.600 | 452.40 | 52.200 | 487.20 | 50.00
06/22 | 609.000 | 535.05 | 73.950 | 609.00 | 41.67
07/22 | 591.600 | 474.15 | 117.450 | 587.25 | 58.82
08/22 | 874.350 | 717.75 | 156.600 | 848.25 | 33.33
09/22 | 800.400 | 591.60 | 208.800 | 704.70 | 33.33
10/22 | 604.650 | 321.90 | 282.750 | 561.15 | 35.42
11/22 | 674.250 | 308.85 | 365.400 | 635.10 | 29.23
12/22 | 1150.575 | 674.25 | 476.325 | 1183.20 | 30.36
Total | 7490.7 | 5694.15 | 1796.55 | 7312.35 |

##  6 Conclusion

This work introduces a novel Bayesian nowcasting model tailored to estimate
IBNR cyber incidents. Through the synthetic and empirical data studies, the
model has demonstrated exceptional predictive accuracy, outperforming
traditional models commonly used in actuarial science, such as the chain-
ladder and generalized linear models. The application of our model in reserve
estimation, utilizing the average cost of a data breach reported by IBM,
offers a refined approach to quantifying the financial implications of cyber
incidents. This not only provides reserves estimations but also contributes
valuable insights into the dynamic nature of cyber risk exposure. The proposed
Bayesian nowcasting model represents a significant advancement in actuarial
science, particularly in the domain of cyber risk management. Its ability to
effectively integrate temporal and correlation effects among delayed reporting
intervals sets a new benchmark for predictive analytics in insurance reserves.

While this study focuses on cyber incidents, the proposed model is adaptable
to other types of insurance claims where reporting delays are a concern. By
integrating more detailed data regarding the specifics of cyber incidents, the
model’s predictive capabilities can be enhanced. For instance, in the context
of reserve estimation, employing actual cost data for each incident, rather
than relying on average costs, would likely yield more precise outcomes.
Nonetheless, obtaining such detailed cost data is often challenging in
practice, presenting a notable limitation to this approach. Furthermore,
exploring a joint modeling approach that simultaneously considers both the
frequency of incidents and their associated costs promises to refine the
model’s performance even further.

## References

  * [1] Glen Barnett and Ben Zehnwirth.  Best estimates for reserves.  In Proceedings of the Casualty Actuarial Society, volume 87, pages 245–321, 2000.
  * [2] Leonardo S Bastos, Theodoros Economou, Marcelo FC Gomes, Daniel AM Villela, Flavio C Coelho, Oswaldo G Cruz, Oliver Stoner, Trevor Bailey, and Claudia T Codeço.  A modelling approach for correcting reporting delays in disease surveillance data.  Statistics in medicine, 38(22):4363–4377, 2019.
  * [3] Ronald L Bornhuetter and Ronald E Ferguson.  The actuary and ibnr.  In Proceedings of the casualty actuarial society, volume 59, pages 181–195, 1972.
  * [4] Stephen P Brooks and Andrew Gelman.  General methods for monitoring convergence of iterative simulations.  Journal of computational and graphical statistics, 7(4):434–455, 1998.
  * [5] Joseph Buckman, J Bockstedt, Matthew J Hashim, and Tiemen Woutersen.  Do organizations learn from a data breach?  In Workshop on the Economics of Information Security, 2017.
  * [6] Perry de Valpine, Daniel Turek, Christopher J Paciorek, Clifford Anderson-Bergman, Duncan Temple Lang, and Rastislav Bodik.  Programming with models: writing statistical algorithms for general model structures with nimble.  Journal of Computational and Graphical Statistics, 26(2):403–413, 2017.
  * [7] Łukasz Delong, Mathias Lindholm, and Mario V Wüthrich.  Collective reserving using individual claims data.  Scandinavian Actuarial Journal, 2022(1):1–28, 2022.
  * [8] Benjamin Edwards, Steven Hofmeyr, and Stephanie Forrest.  Hype and heavy tails: A closer look at data breaches.  Journal of Cybersecurity, 2(1):3–14, 2016.
  * [9] Martin Eling and Nicola Loperfido.  Data breaches: Goodness of fit, pricing, and risk measurement.  Insurance: mathematics and economics, 75:126–136, 2017.
  * [10] Peter England and Richard Verrall.  Analytic and bootstrap estimates of prediction errors in claims reserving.  Insurance: mathematics and economics, 25(3):281–293, 1999.
  * [11] Andrew Gelman and Donald B Rubin.  Inference from iterative simulation using multiple sequences.  Statistical science, 7(4):457–472, 1992.
  * [12] Kazuki Ikegami and Hiroaki Kikuchi.  Modeling the risk of data breach incidents at the firm level.  In International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, pages 135–148. Springer, 2020.
  * [13] Jun S. Liu.  Monte Carlo strategies in scientific computing, volume 75.  Springer, 2001.
  * [14] Thomas Mack.  Distribution-free calculation of the standard error of chain ladder reserve estimates.  ASTIN Bulletin, 23(02):213–225, 1993.
  * [15] Thomas Mack.  The standard error of chain ladder reserve estimates: Recursive calculation and inclusion of a tail factor.  ASTIN Bulletin, 29(2):361–366, 1999.
  * [16] Sarah F. McGough, Michael A. Johansson, Marc Lipsitch, and Nicolas A. Menzies.  Nowcasting by bayesian smoothing: A flexible, generalizable model for real-time epidemic tracking.  PLoS Computational Biology, 16, 2019.
  * [17] A E Renshaw and R J Verrall.  A stochastic model underlying the chain-ladder technique.  British Actuarial Journal, 4(4):903–923, 1998.
  * [18] Sasha Romanosky, Rahul Telang, and Alessandro Acquisti.  Do data breach disclosure laws reduce identity theft?  Journal of Policy Analysis and Management, 30(2):256–286, 2011\.
  * [19] Hong Sun, Maochao Xu, and Peng Zhao.  Modeling malicious hacking data breach risks.  North American Actuarial Journal, 25(4):484–502, 2021.
  * [20] Hong Sun, Maochao Xu, and Peng Zhao.  A multivariate frequency-severity framework for healthcare data breaches.  The Annals of Applied Statistics, 17(1):240–268, 2023.
  * [21] Mario V Wüthrich.  Machine learning in individual claims reserving.  Scandinavian Actuarial Journal, 2018(6):465–480, 2018.
  * [22] Mario V Wüthrich and Michael Merz.  Stochastic claims reserving methods in insurance.  John Wiley & Sons, 2008.
  * [23] Maochao Xu and Quynh Nhu Nguyen.  Statistical modeling of data breach risks: Time to identification and notification.  Variance, to appear (arXiv preprint arXiv:2209.07306), 2022.


# Sampling Through the Lens of Sequential Decision Making

Jason Xiaotian Dou
University of Pittsburgh
<EMAIL_ADDRESS>Alvin Qingkai Pan
Columbia University
<EMAIL_ADDRESS>Runxue Bao
University of Pittsburgh
<EMAIL_ADDRESS>Harry Haiyi Mao
University of Pittsburgh
<EMAIL_ADDRESS>Lei Luo
University of Pittsburgh
<EMAIL_ADDRESS>
###### Abstract

Sampling is ubiquitous in machine learning methodologies. Due to the growth of
large datasets and model complexity, we want to learn and adapt the sampling
process while training a representation. Towards achieving this grand goal, a
variety of sampling techniques have been proposed. However, most of them
either use a fixed sampling scheme or adjust the sampling scheme based on
simple heuristics. They cannot choose the best sample for model training in
different stages. Inspired by “Think, Fast and Slow” (System 1 and System 2)
in cognitive science, we propose a reward-guided sampling strategy called
Adaptive Sample with Reward (ASR) to tackle this challenge. To the best of our
knowledge, this is the first work utilizing reinforcement learning (RL) to
address the sampling problem in representation learning. Our approach
optimally adjusts the sampling process to achieve optimal performance. We
explore geographical relationships among samples by distance-based sampling to
maximize overall cumulative reward. We apply ASR to the long-standing sampling
problems in similarity-based loss functions. Empirical results in information
retrieval and clustering demonstrate ASR’s superb performance across different
datasets. We also discuss an engrossing phenomenon which we name as “ASR
gravity well” in experiments.

##  1 Introduction

Various sampling techniques have been developed to cope with learning
problems’ high complexity and heterogeneity. In this paper, we approach the
sampling problem in a cognitive science-informed way. In cognitive science,
Daniel Kahneman develops a theory of “Thinking, Fast and Slow” to explain
human decision making [1, 2, 3, 4, 5]. In Kahneman’s theory, human decisions
are supported by two main kinds of capabilities. System 1 is guided by
intuition rather than deliberation and gives fast answers to simple questions.
On the contrary, system 2 solves more complex questions with additional
computational resources, full attention, and sophisticated logical reasoning.
Figure 1 gives an overview of the two systems of thinking. A closer
comprehension of how humans have evolved to obtain can inspire innovative ways
to imbue AI with competencies [2]. [1] provides a unified theory aiming to
identify the roots of desired human capabilities.

Current adaptive sampling methods [6, 7, 8, 9, 10] are similar to System 1 and
reinforcement learning [11, 12, 13] is one step closer to System 2. One
representative System 1 sampling method is semihard sampling (which yields
samples that is fairly hard but not too hard) [14]. While as the sample size
grows, relying on relatively simple heuristic based methods to drive learning
becomes challenging. System 2 becomes the better approach. We propose System 2
as a reinforcement learning-based sampling approach where a reward function
guides the sampling process called Adaptive Sampling with Reward (ASR). We
apply the ASR framework to the representation learning sampling problem and
achieve better performance than competitive, commonly adopted sampling methods
(System 1). We observe an engrossing phenomenon called “ASR gravity well” in
the reinforcement learning policy initialization step, which concurs the
fascinating “softmax gravity well” summarized in [15].

Figure 1: Overview of System 1 vs System 2 approaches to sampling: heuristics
based methods like semi-hard sampling and distance based sampling represent
System 1, the ASR approach (Reinforcement learning) we propose represents
System 2. A comprehensive evaluation of different heuristic based methods can
be found in [14]. In Section 4 we give detailed explanation of the ASR
framework.

We organize the rest of the paper as the following: in Section 2, we review
some of the classic and recent work on the applications of reinforcement
learning, the interplay between reinforcement learning and representation
learning, and adaptive sampling methods. Then in Section 3, we detail
reinforcement learning’s basic formulation and the representation learning
task metrics NMI, Recall, and F1, which we use to construct the reward
structure in Section 4. Later in Section 4, we elaborate the System 2 we
design. We first explain how we model the sampling process in representation
learning as a reinforcement learning problem. Then we present the policy
gradient theorem and our way to optimize the policy objective [16]. Section 5
compares the two systems: the ASR approach with the widely used semi-hard and
distance-based sampling strategies on two representation learning tasks,
information retrieval, and clustering. We demonstrate the ASR framework’s
superiority empirically over three benchmark datasets. In Section 5, we
provide a social impact statement and discussion, and finally, in Section 6,
we conclude our research and suggest several intriguing questions for future
research endeavors.

##  2 Related Work

###  2.1 Reinforcement Learning

There has been a flurry of recent work on reinforcement learning methodologies
and applications [17, 18, 19, 20, 21, 22, 23, 24, 25]. One of the most
phenomenal applications of reinforcement learning, which has made a tremendous
impact on the AI community and beyond, is the application of Go [26].
Furthermore, [27] works on RL for unsupervised skill discovery. [28] applies
reinforcement learning with behavior primitives for diverse manipulation
tasks. [29] formulates the protein folding as a reinforcement learning
problem. [30] proposes a framework adaptive reinforcement learning, called
AdaRL, that adapts reliably and efficiently to changes across domains with a
few samples from the target domain, even in partially observable environments.

Above scenarios often consider single agent setting. Recently mean field
theory [31] has been used to aggregate agents to address the multi-agent
reinforcement learning problems [32, 33, 34]. Learning in Multi-agent
reinforcement learning is fundamentally difficult since agents not only
interact with the environment but also with each other. Traditional Q-learning
considers other agents as the environment fails in multi-agent setting and
makes the learning unstable: one agent’s policy change will affect other’s.
Specifically, mean field reinforcement learning addresses the above problem by
learning in a stochastic games with a large number of agents, where the
empirical mean action is used as mean field. And each agent uses the mean
field to update Q-function and policy [35, 33].

###  2.2 Interplay between Reinforcement Learning and Representation Learning

Representation learning allows an AI system to automatically discover the
representations needed for feature detection or classification from raw data.
Due to its effectiveness in real-world tasks, intensive efforts have been put
to design various representation learning algorithms in recent years.
Questions in representation learning include appropriating objectives for
learning good representations, for computing representations (i.e.,
inference), and the geometrical connections between representation learning
[36, 37, 38, 39, 40]. [41] considers learning representations (features) in
the setting in which we have access to multiple unlabeled views of the data
for learning while only one view is available for downstream tasks. While [42]
introduces a self-supervised approach for learning node and graph level
representations by contrasting structural views of graphs. [43] shows that
progress in image generation quality translates to substantially improved
representation learning performance and proposes a large scale adversarial
representation learning approach. In addition, representation learning is
prominent in machine learning applications in computational biology [44, 45,
46], natural language processing [47, 48, 49] and computer vision [50] and
computational social science [51].

Using reinforcement learning in high dimensional data can be arduous and
tedious, where the training may not make significant progress without
excessive hyperparameter tuning. Indeed, conventional reinforcement learning
methods are demonstrated to be restricted by computational and data efficiency
issues without a well-learned representation [52, 53, 54]. Therefore, it’s not
surprising that numerous work has been done in the past that utilize latent
space representations, such as [39, 55, 56, 57, 58, 59].

In particular, stochastic Latent Actor-Critic (SLAC) [55] is an enhanced
method on top of the classic Soft Actor-Critic (SAC) [60] that extends to
partially observed Markov Decision Process (POMDP) [61, 62]. The intuition is
to optimize the joint likelihood of observation (state) space and the
optimality of each time step conditioned on past actions given by maximum
entropy policies. The exact objective is to maximize the evidence lower bound
(ELBO) of the logarithm of the joint likelihood computed from the posterior of
the factorization of the variational distribution. By explicitly learning
latent representations, reinforcement learning from high-dimensional objects
is accelerated, especially for long-horizon continuous control.

###  2.3 Adaptive Sampling

The adaptive sample scheme exploits the specific properties of empirical risk
minimization problems to improve the convergence guarantees for traditional
optimization methods. [6] proposes two types of sample-size schemes: linear
and alternating. Linear means we start from sample size $n$ and perform $2n$
steps. Alternating means every other iteration we sample a new data point,
which is added to the set, and there is also update on the fresh sample.
Afterwards series of work have extend the framework to different optimization
methods and problem settings [6, 7, 8, 9, 10].

##  3 Problem Formulation and Preliminaries

###  3.1 Reinforcement Learning Formulation

In reinforcement learning, a learning agent is embedded in an environment in
such a way that it can discriminate state space $S$ and implement actions in
the action space $\mathcal{A}$[63]. $\forall s\in S$, the agent can choose an
available action $a\in\mathcal{A}$ and gains a immediate reward $r(s,a)$ which
follows a distribution $R$ given $s\in S$ and $a\in\mathcal{A}$. The
environment inherits transition probabilities $P(s^{\prime},r|s,a)$ that
determines the probability of the agent going to $s^{\prime}$ and gains $r$ by
taking $a$ from $s$. Now the objective is to learn an optimal policy where the
remaining decisions must constitute an optimal policy with regard to the state
resulting from the first decisions regardless of the initial state and
decisions [64]. Hence, the agent takes a trajectory with joint probability
distribution given by the optimal policy $\pi^{*}$ so that it maximises the
expected accumulative reward $\mathbb{E}[\sum_{t=0}^{\infty}\eta^{t}r_{t}]$,
where the agent starts at $s_{0}$ and $\eta\in(0,1]$ is the discount factor
[11].

Reinforcement learning and contextual bandits are two widely studied
algorithms for sequential decision making problems. Reinforcement learning is
usually perceived to be more difficult than contextual bandits due to long
horizon and state-dependent transitions. Interestingly, [65] shows that
reinforcement learning poses little additional difficulty on sample complexity
compared to contextual bandit.

###  3.2 Evaluation Metrics as Reward

In representation learning, Recall@n(1,2,4), F1, and Normalized Mutual
Information [50, 14] are benchmarks for clustering and retrieval tasks. We use
them as basis of reward function in reinforcement learning. Thus, we need to
give a brief introduction to these metrics.

  * •

Normalized Mutual Information (NMI) We use NMI to measure the clustering
quality. Normalization of the Mutual Information (MI) score to scale the
results between 0 (no mutual information) and 1 (perfect correlation).

  * •

Recall at k We calculate Recall@k for retrieval tasks. Specifically, for each
query, top-k nearest objects are returned, then the recall score will be
calculated. The same class label as the query positive and the others
negative.

  * •

F1 score The F1-score measures the harmonic mean between precision and recall
which is a commonly used retrieval metric, placing equal importance on both
precision and recall.

##  4 Algorithmic Framework: Adaptive Sampling with Reward (ASR)

###  4.1 Representation Learning and Similarity based Loss Functions

In representation learning, we use $\phi_{i}:=\phi\left(O_{i};\zeta\right)$ to
denote a D-dimensional embedding of an object $O_{i}$ with
$\phi\left(O_{i};\zeta\right)$ being represented by a deep neural network
parameterized by $\zeta$ [50, 55, 14, 66]. This representation makes using
similarity based loss functions as training objectives intuitively.
Contrastive loss, triplet loss [50] and margin loss [66] are among the most
fundamental similarity based loss functions. The contrastive loss minimizes
objects’ distance in the embedding space if their classes are the same, and
separates them a fixed margin away otherwise. The triplet loss takes triplets
of anchor, positive, and negative objects, and enforces the distance between
the anchor and the positive to be smaller than that between the anchor and the
negative. Margin loss further extends the triplet loss by introducing a
dynamic, learnable boundary $\beta$ between positive and negative pairs. The
formation of contrastive loss is as the following. We first have embedding
pairs $\mathcal{P}$, which is sampled from a minibatch of size $b$. The pair
contains an anchor $\phi_{a}$ from class $y_{a}$ and either a positive
$\phi_{p}$ with $y_{a}=y_{p}$ or a negative $\phi_{n}$ from a different class,
$y_{a}\neq y_{n}$. The distance function we utilize is the standard Euclidean
distance $d_{e}(x,y)=\|x-y\|_{2}$. $\gamma$ is the margin which is usually set
to $1$. Then the network $\phi$ is trained to minimize:

$\mathcal{L}_{\textrm{contrastive}}=\frac{1}{b}\sum_{(i,j)\in\mathcal{P}}^{b}\mathbb{I}_{y_{i}=y_{j}}d_{e}\left(\phi_{i},\phi_{j}\right)+\mathbb{I}_{y_{i}\neq
y_{j}}\left[\gamma-d_{e}\left(\phi_{i},\phi_{j}\right)\right]_{+}$ (1)

Triplets extend the contrastive formulation by providing a triplet
$\mathcal{T}$ sampled from a mini-batch:

$\mathcal{L}_{\textrm{triplet}}=\frac{1}{b}\sum_{(a,p,n)\in\mathcal{T}\atop
y_{a}=y_{p}\neq
y_{n}}^{b}\left[d_{e}\left(\phi_{a},\phi_{p}\right)-d_{e}\left(\phi_{a},\phi_{n}\right)+\gamma\right]_{+}$
(2)

Margin loss transfers the common triplet ranking problem to a relative
ordering of pairs:

$\mathcal{L}_{\textrm{margin}}=\sum_{(i,j)\in\mathcal{P}}\gamma+\mathbb{I}_{y_{i}=y_{j}}\left(d_{e}\left(\phi_{i},\phi_{j}\right)-\beta\right)-\mathbb{I}_{y_{i}\neq
y_{j}}\left(d_{e}\left(\phi_{i},\phi_{j}\right)-\beta\right)$ (3)

Given the above formulations, choosing samples becomes the critical step for
optimal performance, thus why we introduce reinforcement learning sampling
method ASR.

  * •

State Space We use state $s\in S$ to represent current embedding $\phi$. For
generalization performance, we use a weighted sum over Recall, NMI on the
validation set $\mathcal{O}_{\mathrm{val}}$ to describe $s$.

  * •

Action Space and Optimal Policy We build the action space based on the idea of
distance based sampling [14]. [14] proposes to sample triplets from a static
uniform distribution over distance between anchor and negative object
$d_{an}$: $O_{n}\sim p(O_{n}|O_{a})$. Then we construct a discrete set of
parameters to multiply on the probabilities. In different training state $s$,
the optimal policy $\pi_{\theta}(a\mid s)$ controls the adjustment to apply.

  * •

Reward Function A reward function is a function that provides a numerical
score based on the state of the environment. It maps each state of the
environment to a single number, specifying the intrinsic desirability of that
state. As we mentioned in Section 3.1, maximizing the reward is the goal in a
reinforcement learning algorithm. Formally, we have the The Reward Hypothesis:
all of what we mean by goals can be thought as the maximization of the
expected value of the cumulative sum of a received scalar reward [12]. We
compute the reward $r$ for action $a\sim\pi_{\theta}(a\mid s)$ by directly
calculating the difference of $\phi(\cdot;\zeta)$ over
$\phi\left(\cdot;\zeta^{\prime}\right)$ from previous training state.

  * •

Cumulative Reward and Policy Objective In the last part, we model the reward
$r$ in an incremental way, however, the goal is to optimize the overall
cumulative reward. Thus we introduce the total expected reward
$R(\tau)=\sum_{t}r_{t}$. Here $\tau$ is a sequence of training triple for
$t=0,1...,T$. We use $G(s,a)$ to denote the discounted reward, then we can use
policy gradient to optimize the policy objective $J(\theta)$ [67] as

$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}(\tau)}\left[\nabla_{\theta}\log\pi_{\theta}(a\mid
s)G(s,a)\right]$ (4)

Computing the gradient $J(\theta)$ is tricky since it depends on both the
action space and the final stationary distribution of state space. The
immediate policy gradient learning algorithm is the REINFORCE algorithm
(vanilla policy gradient) [12, 11]. But REINFORCE has two glaring issues:
noisy gradients and high variance. Proximal Policy Optimization (PPO) [68]
addresses the above issues by using a surrogate objective to represent the
policy objective $J(\theta)$. Thus we use PPO as our premier optimization
method but also compare it with REINFORCE in the empirical studies.

Figure 2: Ablation on initial distribution on CUB dataset for six different
benchmarks. We observe the “ASR gravity well” from the “Normal_high” curves.

The ASR framework is as the following. First, the sampling distribution
$p(O_{n}|O_{a})$ is initialized with an initial distribution: random,
distance, or Gaussian distribution. Second, we initialize both the adjustment
policy $\pi_{\theta}$ and the auxiliary policy $\pi_{\theta_{\mathrm{old}}}$
for estimating the PPO probability ratio. Then training is performed with
triplets with random anchor-positive pairs and sampled negatives from the
sampling distribution. After selected iterations, reward and state metrics are
computed on the embeddings $\phi(\cdot;\zeta)$ of $\mathcal{O}_{val}$. These
values are aggregated in a training reward $r$ and input state $s$. Then $r$
is used to update the current policy $\pi_{\theta}$ and $s$. Then
$\pi_{\theta}$ and $s$ are fed into the updated policy to estimate adjustments
$a$ for the sampling distribution $p(O_{n}|O_{a})$. After certain iterations
$\pi_{\theta_{\mathrm{old}}}$ is updated with current policy $\pi_{\theta}$.

To summarize, for the System 2 approach, we present an adaptive sampling
setting for adjustments which directly facilitate maximal representation
learning performance. We show how to learn a policy which effectively alters
sampling distribution to optimally support learning of the representation
models.

Distribution |  parameters
---|---
uniform low |  $\mu=0.5,\sigma=0.2$
uniform high |  $\mu=1.5,\sigma=0.2$
distance |  limit of distance $=0.5$
random |  random
normal high |  $\mu=1.6,\sigma=0.04$
normal low |  $\mu=0.5,\sigma=0.05$
Table 1: Initial distribution setup in Section 5.3

##  5 Experiments

We compare the ASR framework with the most popular heuristic based sampling
strategies: semi-hard [50], and distance based sampling [14] with both margin
and triplet losses on the tasks of retrieval and clustering. For optimization
methods, we use both PPO and REINFOCE. As we states earlier, Recall, NMI, and
F1 are utilized as evaluation metrics. In this section, we first present
implementation details, then the benchmark datasets, experimental results,
ablations studies, and in the end, quantitative analysis of our approach to
information retrieval and clustering tasks.

###  5.1 Implementation Details

In line with standard practices [14] we resize and crop images to 224 × 224
for training and center crop to the same size for evaluation. Weight decay is
set to a constant value of $4\times 10^{-4}$. All the models are implemented
in Pytorch. Experiments are performed on individual Nvidia Titan XP GPU with
12 GB memory. Learning rates are set to $10^{-5}$. Policy $\pi$ is implemented
as a two-layer fully-connected network with ReLU-nonlinearity. We choose
triplet with $\gamma=0.2$ and margin $\beta=0.6$. Each training is run over 40
epochs if not specified. We follow the training protocol with ResNet50 [69]
and a $15\%$ subset of training data is used as validation set.

###  5.2 Benchmark Datasets

We use three common benchmark datasets, including CUB200-2011 (CUB) [70],
Cars196 [71] and Stanford Online Products (SOP) [72]. Specifically, CUB
dataset has 11,788 images, which belong to 200 bird classes. CARS196 dataset
has 16,185 images from 196 car classes. SOP dataset contains 120,053 product
images within 22,634 classes. For each dataset, the first half of classes are
used for training, and the other half are used for testing. And a random
subset of $15\%$ of training sets is used as validation.

Figure 3: Ablation on number of epoch on CUB dataset for Section 5.3 ASR on
CUB200-2011 [70]
---
Initial Distribution | R@1 | R@2 | R@4 | R@8 | NMI | F1 | Loss
uniform low | 0.5864 | 0.7129 | 0.8026 | 0.8952 | 0.6539 | 0.3434 | 15.084
uniform high | 0.543 | 0.6723 | 0.7846 | 0.866 | 0.6154 | 0.2912 |  1.084
normal low | 0.6063 | 0.7263 | 0.8255 | 0.8982 | 0.6629 | 0.3571 | 14.093
normal high | 0.5709 | 0.6918 | 0.799 | 0.874 | 0.6367 | 0.319 | 7.856
random | 0.5532 | 0.6768 | 0.7862 | 0.8702 | 0.6174 | 0.2864 | 1.365
distance | 0.5537 | 0.6918 | 0.8011 | 0.0881 | 0.6405 | 0.3296 | 19.967
Table 2: Ablation on initial distribution for CUB200-2011. Training is done
over 40 epochs. We can see “normal high” is the best initial distribution in
most cases. CUB200-2011 [70]
---
Approach | R@1 | R@4 | NMI | F1
ASR(PPO) + Triplet | 0.6063 | 0.8255 | 0.6629 | 0.3571
ASR(PPO) + Margin | 0.604 | 0.8214 | 0.6594 | 0.3557
ASR(REINFORCE) + Triplet | 0.5037 | 0.7562 | 0.6153 | 0.296
Semihard + Triplet | 0.5947 | 0.8067 | 0.6533 | 0.3412
Distance + Triplet | 0.5581 | 0.7942 | 0.6262 | 0.299
CARS196 [71]
| R@1 | R@4 | NMI | F1
ASR(PPO) + Triplet | 0.7150 | 0.8930 | 0.5993 | 0.2925
ASR(PPO) + Margin | 0.6179 | 0.839 | 0.5552 | 0.2627
ASR(REINFORCE) + Triplet | 0.0.7145 | 0.8901 | 0.602 | 0.3104
Semihard + Triplet | 0.6633 | 0.8491 | 0.5836 | 0.2665
Distance + Triplet | 0.6337 | 0.8393 | 0.5508 | 0.2216
SOP [72]
| R@10 | R@100 | NMI | F1
ASR(PPO) + Triplet | 0.9447 | 0.8737 | 0.8914 | 0.3258
ASR(PPO) + Margin | 0.8619 | 0.9387 | 0.8911 | 0.3247
ASR(REINFORCE) + Triplet | 0.85 | 0.9296 | 0.8904 | 0.3217
Semihard + Triplet | 0.8703 | 0.9444 | 0.8916 | 0.3265
Distance + Triplet | 0.8337 | 0.926 | 0.8813 | 0.2806
Table 3: Comparison of sampling strategies and loss functions. Training is
done over 40 epochs. Three datasets are used for comprehensive empirical
evaluation. Four benchmarks are used to evaluate models’ performance. We can
see in most cases ASR approach achieves the best performance.

###  5.3 Results and Analysis

Empirical results in Table 3 demonstrate that reinforcement learning approach
to the adaptive sampling effectively supports better learning of the
representations and achieves superior performance on information retrieval and
clustering tasks. We can see ASR approach outperforms the heuristic-based
sampling approach in all three benchmark datasets.

[15] suggests that optimizing with respect to the softmax exhibits sensitivity
to parameter initialization. So we explore different distribution
initialization and study the impact of the number of epoch in the following
subsections on the CUB dataset. We utilize a variety of distributions named
“uniform”, “distance”, “random”, and “normal” to study policy initiation’s
impact on ASR framework’s performance. Table 1 contains detailed distribution
parameters setting.

####  5.3.1 Policy Initialization: ASR Gravity Well

Measurement | F1 | R@1 | R@2 | R@4 | R@8 | NMI
---|---|---|---|---|---|---
Epoch | 28 | 46 | 46 | 51 | 32 | 28
Table 4: Statistics of the optimal number of epoch on CUB dataset. Training is
done over 150 epochs.

We notice a captivating phenomenon which we name as “ASR gravity well” in the
experiments. “Softmax gravity well”(SGW) [15] describes such a phenomenon:
gradient ascent trajectories are drawn toward suboptimal corners of the
probability simplex and subsequently slowed in their progress toward the
optimal vertex. These facts are observed through empirical observations in
one-state MDPs, Four-room environment, and MNIST dataset in [15], revealing
that the behavior of Softmax policy gradient depends strongly on
initialization. We observe a similar phenomenon in our experiments with
“relative high” normal distribution. When we closely investigate the
“Normal_high” curves in Figure 2, we find they all fall into a global minimum
at around epoch 15. The “reverse peaks” are very significant within five
(Recall@1, Recall@2, Recall@4, Recall@8, and NMI) out of six benchmarks. That
means the phenomenon arises for both information retrieval and clustering
applications. Since ASR adopts softmax policy gradient, we name it as “ASR
gravity well”. “Softmax gravity well” [15] captures that poorly initialized
softmax policy gradient algorithm can get stuck at suboptimal policy for
single-state MDP. We confirm and extend this intriguing observation to
practical deep reinforcement learning’s application in the sampling problem of
representation learning.

For the normal distribution, when the $\mu$ is high, it tends to change
dramatically, so it has a high tendency to trap into the “ASR gravity well”.
We also notice “normal low” and “uniform low” are the two distributions that
perform the best from Table 2. It’s intuitive since then we have smaller
$\sigma$, a normal distribution skew towards uniform distribution. And actions
with larger true rewards will have larger gradients in normal low/uniform
distribution therefore the attraction towards suboptimal corners will not
happen. It would be interesting to see how the phenomenon evolves in other
scenarios like attention models and exponential exploration.

####  5.3.2 Number of Epoch’s Impact

We study the optimal number of epochs to train with CUB dataset by ASR
framework and triplet loss. The curves of experimental results are shown in
Figure 3 and the numerical performances are summarized in Table 4. We find our
method usually achieves the best performance between 30 epochs and 50 epochs.
This confirms our practice that different comparison methods all run 40 epochs
to reach (approximate) optimum in experiments. Increasing number of epochs can
cause the over-fitting problem and deteriorate the performance.

##  6 Social Impact and Discussion

This research pursues methodology innovation, so our social implications are
relatively high level and have no direct negative impact. While machine
learning technologies have consumed significant energy footprint, our research
can increase energy consumption.

Reinforcement learning can help guiding sampling process, however it increases
the total number of hyper-parameters required to be tuned. This can be a
drawback in comparison to simpler methods like (contextual) bandit and active
learning. So ASR framework is the best suitable in machine learning settings
where the perspective of long term return instead of the immediate reward
(active learning) is critical. We are glad to validate empirically that the
representation learning sampling problem is the case.

##  7 Conclusion

We propose the ASR framework first applying reinforcement learning to address
adaptive sampling problem in representation learning. The ASR framework
represents System 2 Thinking in cognitive science theory and heuristic-based
baseline methods represent System 1 thinking. Experiments in representation
learning show that ASR with triplet loss has the best performance for most
metrics on two applications and three datasets.

The competitive performance across different datasets validate our method’s
generalizability and robustness. Additionally, our work leaves several
intriguing questions: natural policy gradient and log-barrier regularization
offer nice theoretical convergence properties, how do they compare to proximal
policy optimization in the sampling framework [73]? How much “escape time”
around each suboptimal corner does the algorithm spend to be out of the
“gravity well”? Other interesting directions are extending the sampling
framework to other areas like few-shot Meta-Learning, and zero-shot Domain
Adaptation.

## References

  * [1] Daniel Kahneman.  Thinking, fast and slow.  Macmillan, 2011.
  * [2] Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jon Lenchner, Nick Linck, Andrea Loreggia, Keerthiram Murugesan, Nicholas Mattei, Francesca Rossi, and Biplav Srivastava.  Thinking fast and slow in ai, 2020.
  * [3] Sandhya Saisubramanian, Ece Kamar, and Shlomo Zilberstein.  A multi-objective approach to mitigate negative side effects.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 354–361, 2021\.
  * [4] Jason Dou.  Impartial redistricting: a markov chain approach to the "gerrymandering problem", 2017.
  * [5] Lucy Chenyun Wu, Jason Xiaotian Dou, Danny Sleator, Alan Frieze, and David Miller.  Impartial redistricting: A markov chain approach.  arXiv preprint arXiv:1510.03247, 2015.
  * [6] Hadi Daneshmand, Aurelien Lucchi, and Thomas Hofmann.  Starting small-learning with adaptive sample sizes.  In ICML, pages 1463–1471, 2016.
  * [7] Aryan Mokhtari, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, and Alejandro Ribeiro.  Adaptive newton method for empirical risk minimization to statistical accuracy.  In Neurips, pages 4062–4070, 2016.
  * [8] Aryan Mokhtari and Alejandro Ribeiro.  First-order adaptive sample size methods to reduce complexity of empirical risk minimization.  In Neurips, pages 2060–2068, 2017.
  * [9] Lihua Lei and Michael I Jordan.  Less than a single pass: Stochastically controlled stochastic gradient method.  arXiv preprint arXiv:1609.03261, 2016.
  * [10] Aryan Mokhtari, Asuman Ozdaglar, and Ali Jadbabaie.  Efficient nonconvex empirical risk minimization via adaptive sample size methods.  In AISTATS, pages 2485–2494, 2019.
  * [11] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun.  Reinforcement learning: Theory and algorithms.  CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 2019.
  * [12] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [13] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine.  Dr3: Value-based deep reinforcement learning requires explicit regularization.  arXiv preprint arXiv:2112.04716, 2021.
  * [14] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl.  Sampling matters in deep embedding learning.  In ICCV, 2017.
  * [15] Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans.  Escaping the gravitational pull of softmax.  Neurips, 33, 2020.
  * [16] Ying Jin, Zhuoran Yang, and Zhaoran Wang.  Is pessimism provably efficient for offline rl?  In International Conference on Machine Learning, pages 5084–5096. PMLR, 2021.
  * [17] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine.  Parrot: Data-driven behavioral priors for reinforcement learning.  In ICLR, 2021.
  * [18] Zebang Shen, Alejandro Ribeiro, Hamed Hassani, Hui Qian, and Chao Mi.  Hessian aided policy gradient.  In ICML, pages 5729–5738, 2019.
  * [19] Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Başar.  Near-optimal regret bounds for model-free rl in non-stationary episodic mdps.  arXiv preprint arXiv:2010.03161, 2020.
  * [20] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Başar.  Global convergence of policy gradient methods to (almost) locally optimal policies.  arXiv preprint arXiv:1906.08383, 2019.
  * [21] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang.  Neural policy gradient methods: Global optimality and rates of convergence.  arXiv preprint arXiv:1909.01150, 2019.
  * [22] Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell.  Regularization matters in policy optimization - an empirical study on continuous control.  In ICLR, 2021.
  * [23] Lisa Lee, Benjamin Eysenbach, Ruslan Salakhutdinov, Shixiang Shane Gu, and Chelsea Finn.  Weakly-supervised reinforcement learning for controllable behavior, 2020\.
  * [24] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang.  Does unsupervised architecture representation learning help neural architecture search?, 2020.
  * [25] Yixin Wang and Michael I. Jordan.  Desiderata for representation learning: A causal perspective, 2021.
  * [26] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [27] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel.  CIC: Contrastive intrinsic control for unsupervised skill discovery.  In Deep RL Workshop NeurIPS 2021, 2021.
  * [28] Soroush Nasiriany, Huihan Liu, and Yuke Zhu.  Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks.  In Deep RL Workshop NeurIPS 2021, 2021.
  * [29] Wenhao Gao et al.  Development of a Protein Folding Environment for Reinforcement Learning.  PhD thesis, Johns Hopkins University, 2020.
  * [30] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang.  Adarl: What, where, and how to adapt in transfer reinforcement learning, 2021.
  * [31] Henrik Flyvbjerg, Kim Sneppen, and Per Bak.  Mean field theory for a simple model of evolution.  Physical review letters, 71(24):4087, 1993.
  * [32] Sriram Ganapathi Subramanian, Matthew E. Taylor, Mark Crowley, and Pascal Poupart.  Partially observable mean field reinforcement learning, 2021.
  * [33] Sriram Ganapathi Subramanian, Matthew E Taylor, Mark Crowley, and Pascal Poupart.  Decentralized mean field games.  arXiv preprint arXiv:2112.09099, 2021.
  * [34] Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E. Taylor, and Nidhi Hegde.  Multi type mean field reinforcement learning, 2022.
  * [35] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang.  Mean field multi-agent reinforcement learning, 2020.
  * [36] Haiyi Mao, Minxue Jia, Jason Xiaotian Dou, Haotian Zhang, and Panayiotis V. Benos.  Coem: Cross-modal embedding for metacell identification, 2022.
  * [37] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives, 2014.
  * [38] Denis Maua and Cassio De Campos.  Anytime marginal map inference.  arXiv preprint arXiv:1206.6424, 2012.
  * [39] Jason Xiaotian Dou, Lei Luo, and Raymond Mingrui Yang.  An optimal transport approach to deep metric learning (student abstract).  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12935–12936, 2022.
  * [40] Jason Xiaotian Dou, Michelle Liu, Haaris Muneer, and Adam Schlussel.  What words do we use to lie?: Word choice in deceptive messages, 2017\.
  * [41] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes.  On deep multi-view representation learning.  In International conference on machine learning, pages 1083–1092. PMLR, 2015.
  * [42] Kaveh Hassani and Amir Hosein Khasahmadi.  Contrastive multi-view representation learning on graphs.  In International Conference on Machine Learning, pages 4116–4126. PMLR, 2020.
  * [43] Jeff Donahue and Karen Simonyan.  Large scale adversarial representation learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [44] Mario Flores, Zhentao Liu, Ting-He Zhang, Md Musaddaqui Hasib, Yu-Chiao Chiu, Zhenqing Ye, Karla Paniagua, Sumin Jo, Jianqiu Zhang, Shou-Jiang Gao, Yu-Fang Jin, Yidong Chen, and Yufei Huang.  Deep learning tackles single-cell analysis a survey of deep learning for scrna-seq analysis, 2021.
  * [45] Chuizheng Meng, Loc Trinh, Nan Xu, and Yan Liu.  Mimic-if: Interpretability and fairness evaluation of deep learning models on mimic-iv dataset, 2021.
  * [46] Romain Lopez, Jeffrey Regier, Michael Cole, Michael Jordan, and Nir Yosef.  A deep generative model for gene expression profiles from single-cell rna sequencing, 2018.
  * [47] Simon Ståhlberg, Guillem Francès, and Jendrik Seipp.  Learning generalized unsolvability heuristics for classical planning.  In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4175–4181. International Joint Conferences on Artificial Intelligence Organization, 8 2021.  Main Track.
  * [48] Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, and Claire Cardie.  Improving machine reading comprehension with contextualized commonsense knowledge, 2020.
  * [49] Olga Majewska, Charlotte Collins, Simon Baker, Jari Björne, Susan Windisch Brown, Anna Korhonen, and Martha Palmer.  Bioverbnet: a large semantic-syntactic classification of verbs in biomedicine.  Journal of Biomedical Semantics, 12(1):1–13, 2021.
  * [50] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In CVPR, pages 815–823, 2015.
  * [51] Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu.  Identifying Coordinated Accounts on Social Media through Hidden Influence and Group Behaviours, page 1441–1451.  Association for Computing Machinery, New York, NY, USA, 2021.
  * [52] Vik Goel, Jameson Weng, and Pascal Poupart.  Unsupervised video object segmentation for deep reinforcement learning, 2018.
  * [53] Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell.  Loss is its own reward: Self-supervision for reinforcement learning.  arXiv preprint arXiv:1612.07307, 2016.
  * [54] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling, 2021\.
  * [55] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  arXiv preprint arXiv:1907.00953, 2019.
  * [56] Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin.  A framework for efficient robotic manipulation.  arXiv preprint arXiv:2012.07975, 2020.
  * [57] Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine.  Visual reinforcement learning with imagined goals.  arXiv preprint arXiv:1807.04742, 2018.
  * [58] Dibya Ghosh, Abhishek Gupta, and Sergey Levine.  Learning actionable representations with goal-conditioned policies.  arXiv preprint arXiv:1811.07819, 2018.
  * [59] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare.  Deepmdp: Learning continuous latent space models for representation learning.  In ICML, pages 2170–2179. PMLR, 2019.
  * [60] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In ICML, pages 1861–1870. PMLR, 2018.
  * [61] Ricardo Salmon and Pascal Poupart.  On the relationship between satisfiability and markov decision processes.  In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 1105–1115. PMLR, 22–25 Jul 2020\.
  * [62] Pascal Poupart and Craig Boutilier.  Vector-space analysis of belief-state approximation for pomdps, 2013.
  * [63] Leslie Pack Kaelbling.  Learning to achieve goals.  In IJCAI, pages 1094–1099. Citeseer, 1993.
  * [64] Richard Bellman.  The theory of dynamic programming.  Technical report, Rand corp santa monica ca, 1954.
  * [65] Zihan Zhang, Xiangyang Ji, and Simon Du.  Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon.  In COLT, pages 4528–4531. PMLR, 2021.
  * [66] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, and Joseph Paul Cohen.  Revisiting training strategies and generalization performance in deep metric learning, 2020.
  * [67] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang.  Convergent policy optimization for safe reinforcement learning.  Neurips, 32:3127–3139, 2019.
  * [68] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [69] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [70] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [71] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In (3dRR-13), Sydney, Australia, 2013.
  * [72] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In CVPR, 2016.
  * [73] Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang.  Provably efficient exploration in policy optimization.  In International Conference on Machine Learning, pages 1283–1294. PMLR, 2020.