# Blind Number Sequencing

Parker Kuklinski  Nick Vogel

(October 2023)

###### Abstract

A popular challenge on the social media app TikTok is to place the output of a
random number generator in ascending order without arriving at a
contradiction. Most players rely on intuition to construct their sequences and
debate has surfaced in the comments about the optimal strategy to win the
game. To this end, we determine an optimal strategy that offers a 21%
improvement in winning odds over the intuitive “equal-spacing” strategy in the
popular $\\#$20numberchallenge variant.

##  1 Introduction

TikTok is a video hosting social media platform owned by ByteDance [1], and
according to similarweb was the $14^{\text{th}}$ most popular website on the
internet in October 2023 [2]. Through certain design choices, The platform has
curated a unique culture that encourages users to follow trends, often denoted
with the hashtag symbol, #. Certain trends are facilitated through the use of
‘filters’, or video overlays that warp faces, modify voices, or even act as
interactive games. One simple filter that became popular is the random number
filter, which at the tap of the screen, displays a random number between 0-999
over the user’s head. Many use the filter as an oracle to answer humorous
questions (e.x. How many years do I have left to live? How many push-ups
should I do right now? How many kids will I have?)

One user eventually invented the 5-number challenge, played as follows: the
user begins with an empty list of five items, generates a random number, and
places that number somewhere in the list. The user then continues to generate
random numbers one at a time and placing them in empty slots in the list _in
ascending order_ until either the number cannot fit in any available slots or
all slots have been filled. This would sometimes be referred to as ‘blind
number sequencing’ since the player is attempting to sequence numbers one at a
time without knowing what comes next. The game was conceptually simple,
entertaining to watch, easy to play along, and thus primed to become a popular
trend in the community.

As users started beating the 5-number challenge with ease (win probability
with optimal strategy is $\approx 22\%$), participants considered larger
number games, eventually coalescing around the 20-number challenge. To date,
only a handful of players have ever succeeded in completing the challenge. Due
to the difficulty of the challenge and mass of viewers, debate over optimal
strategies raged in the comments of these $\\#$20numberchallenge videos; some
argued that placing numbers based on evenly dividing the interval was optimal,
others advocated for riskier maneuvers to remain in the game longer.

Fortunately the $n$-number challenge is a probabilistic system that can be
analytically modeled; with some effort optimal strategies and exact win
probabilities can be computed. In particular, the probability of winning the
20-number challenge with an optimal strategy is just under $1/8000$. It turns
out that this optimal strategy is not to evenly divide the interval as some
suggested, but to be slightly more avoidant of placing numbers at the ends of
available bins.

The remainder of the paper is structured as follows: section 2 formally
defines an $n$-number game and what is meant by a strategy. Section 3 presents
the equal-spacing strategy while section 4 documents the optimal risk-tolerant
strategy. Section 5 notes a practical consideration of implementing these
strategies that slightly tempers the advantage of the risk-tolerant strategy.

##  2 Definitions and Methods

Let $n\in\mathbb{N}$. In the $n$-_number game_ , a random sequence ${\bf
x}\in\\{000,...,999\\}^{n}$ (not necessarily in ascending order) is revealed
to the player one element at a time, who likewise defines a function
$\sigma:{\bf x}\rightarrow\\{1,...,n\\}$ one element at a time. The player
_wins_ if $\sigma$ is a monotonically increasing bijection. To ease the
analysis in this paper, we consider a variant where the elements ${\bf
x}\in[0,1]^{n}$ are uniformly distributed random variables. We say a
_strategy_ $\sigma=\\{\sigma_{1},...,\sigma_{n}\\}$ is a collection of
functions $\sigma_{k}:[0,1]\rightarrow\\{1,...,k\\}$ that each map a random
number to an entry on a length $k$ list. These strategies can be represented
as nondecreasing step functions such that for each strategy function
$\sigma_{k}$ we can associate a collection of numbers
$\\{\alpha_{k,0},...,\alpha_{k,k}\\}$ such that if
$x\in(\alpha_{k,j-1},\alpha_{k,j})$, then $\sigma_{k}(x)=j$ (we necessarily
have $\alpha_{k,0}=0$ and $\alpha_{k,k}=1$).

The objective of the $n$-number game is to then define a strategy $\sigma$ (or
equivalently a collection of $\\{\alpha_{k,j}\\}_{1\leq j\leq k\leq n}$) which
maximizes the probability $p_{n}$ of correctly blind-sequencing an $n$-length
random sequence ${\bf x}$. Suppose $x\in{\bf x}$ is the first random number
presented in the list. If we place this number in the $\sigma_{n}(x)$ slot in
our empty sequence, then the probability we can correctly blind sequence the
rest of the numbers in ${\bf x}$ is equal to the probability that
$\sigma_{n}(x)$ is the correct placement of $x\in{\bf x}$ (otherwise that
there are $\sigma_{n}(x)-1$ elements of ${\bf x}$ less than $x$ and
$n-\sigma_{n}(x)$ elements of ${\bf x}$ greater than $x$) times the
probability that we will be able to sequence those uniformly distributed
random elements as they are presented to us one at a time (i.e. times
$p_{\sigma_{n}(x)-1}p_{n-\sigma_{n}(x)}$). Integrating $x$ across the domain
$[0,1]$ gives a recursive convolution-type definition of $p_{n}$ given a
strategy $\tilde{\sigma}$:

$p_{n}=\int_{0}^{1}{n-1\choose\sigma_{n}(x)-1}x^{\sigma_{n}(x)-1}(1-x)^{n-\sigma_{n}(x)}p_{\sigma_{n}(x)-1}p_{n-\sigma_{n}(x)}dx$

Here we adopt convention that $p_{0}=1$. By recognizing the piecewise
structure of $\sigma_{n}(x)$ we can clean up the notation as follows:

$p_{n}=\sum_{k=1}^{n}{n-1\choose
k-1}p_{k-1}p_{n-k}\int_{\alpha_{n,k-1}}^{\alpha_{n,k}}x^{k-1}(1-x)^{n-k}dx$
(1)

It may be tempting to attempt to collect these probabilities in a generating
function $p(z)=\sum_{n=0}^{\infty}p_{n}z^{n}$ and attempt derive a formula for
$p(z)$ using Gosper-Zeilberger algorithm [3], however a closed form is
unreachable.

##  3 Equal-Spacing Strategy

Without explicitly calculating optimal values of $\alpha_{k,j}$ (i.e.
maximizing $p_{n}$), one may naively assume that an equal-spacing strategy
would be optimal, or that $\alpha_{k,j}=j/k$. One can show that this strategy
is greedy in the sense that it maximizes the probability that a sequence in
progress is correct _so far_. Let $v_{n}\in\\{[0,1]\cup\emptyset\\}^{n}$
denote an $n$-length sequence in progress where if $v_{n}$ has $k$ nonempty
entries with corresponding inputs in ${\bf x}$, this represents the current
state of an $n$-number game at the $k^{\text{th}}$ turn. We can then define a
function $\tilde{p}:\\{[0,1]\cup\emptyset\\}^{n}\rightarrow[0,1]$ such that
$\tilde{p}(v_{n})$ is the probability that $v_{n}$ correctly orders its
nonempty elements in ${\bf x}$. If the nonempty elements of $v_{n}$ partition
$\\{1,...,n\\}$ into $j$ ”bins” of consecutive empty elements, let $n_{i}$ be
the size of the $i^{\text{th}}$ bin and let $p_{i}$ be the probability that a
random element $x\in[0,1]$ will fit into the $i^{\text{th}}$ bin. Then
$\tilde{p}(v_{n})$ is simply the probability that the remaining $n-k$ elements
of ${\bf x}$ can be placed in the $j$ empty bins, otherwise

$\tilde{p}(v_{n})={n_{1}+\ldots+n_{j}\choose
n_{1},...,n_{j}}p_{1}^{n_{1}}\cdot\ldots\cdot p_{j}^{n_{j}}$

We can prove that an equal-spacing strategy $\sigma^{ES}$ maximizes
$\tilde{p}$ for every new element $x$ appended to $v_{n}$. Due to the product
structure of $\tilde{p}(v_{n})$ and since each new $x$ can only fit in one of
the $j$ empty bins, it suffices to prove optimality for the fully empty vector
$v_{n}=\emptyset$. If we are presented with a new random element $x\in[0,1]$,
let $v_{n}(x)$ be the updated vector with $x$ in the $\sigma_{n}(x)=k$ slot.
Then we have

$\tilde{p}(v_{n}(x))={n-1\choose k-1}x^{k-1}(1-x)^{n-k}$

For shorthand, let $\tilde{p}_{k}$ be this probability corresponding to
selecting $\sigma_{n}(x)=k$. Then we have

$\frac{\tilde{p}_{k+1}}{\tilde{p}_{k}}=\frac{(n-k)x}{k(1-x)}$

It follows that $\tilde{p}_{k+1}>\tilde{p}_{k}$ when $x>k/n$ and
$\tilde{p}_{k}>\tilde{p}_{k+1}$ when $x<k/n$. Therefore $\tilde{p}_{k}$ is
maximized when we choose $x\in(\frac{k-1}{n},\frac{k}{n})$.

##  4 Risk-Tolerant Strategy

While the equal-spacing strategy aligns with intuition and greedily maximizes
the probability that the current vector is correct, it does not take into
account the probability that one will be able sequence the remaining elements
as they are presented. In this section, we will demonstrate that in certain
cases, selecting a less likely placement of $x\in{\bf x}$ is countered by an
improvement in the probability of being able to order the remaining elements
of ${\bf x}$. Adopting these modified strategies results in small gains in
$p_{n}$ for $n$ small, but significant gains as $n$ increases.

Let us first demonstrate a superior strategy to equal-spacing in the case of
$n=3$. It is easy to show that a symmetric selection criteria for $n=2$ (i.e.
$\alpha_{2,1}=1/2$) is still optimal with $p_{2}=3/4$. For $n=3$ let
$\alpha_{3,1}=1-\alpha_{3,2}=\alpha$. It can be shown that for an equal-
spacing strategy $\alpha=1/3$ we have $p_{3}=83/162\approx 0.5123$. However
revisiting (1) we can rewrite $p_{3}$ in terms of variable $\alpha$:

$p_{3}=\frac{11}{6}a^{3}-\frac{7}{2}a^{2}+\frac{3}{2}a+\frac{1}{3}$

From this formula we see that $p_{3}$ attains a maximum value of
$p_{3}=377/726\approx 0.5193$ when $\alpha=3/11$. Intuitively speaking, rather
than maximizing the probability that the random number $x$ belongs in slot
$k$, we sacrifice some of that probability so that if $x$ is indeed in slot
$k$, sequencing the rest of the numbers will be easier. In the case of the
3-number game, we are more inclined to place the first random number $x$ in
the $2^{nd}$ window because, if correct, the remaining numbers are
automatically sequenced depending on whether they lie below or above $x$. This
improvement in odds of winning over the equal-spacing strategy is relatively
small for the 3-number game, however we will see that the improvement becomes
more significant as $n$ increases.

Fortunately, there is a readily apparent visual intuition to selecting the
optimal strategy for larger $n$. Revisiting EQN, let us define the function

$f_{n,k}(x)={n-1\choose k-1}p_{k-1}p_{n-k}x^{k-1}(1-x)^{n-k}$ (2)

or the probability of correctly sequencing $n$ uniform random variables on
$[0,1]$ given that you have placed the first random variable $x$ in the
$k^{\text{th}}$ slot. From the piecewise structure of (1), it is evident that
in order to maximize $p_{n}$, we need to select $\\{a_{n,k}\\}$ such that for
$x\in(\alpha_{n,k-1},\alpha_{n,k})$, $f_{n,k}(x)>f_{n,j}(x)$ for all $j\neq
k$. This is depicted in Figure 1 as we plot all of the functions $f_{n,k}(x)$
for fixed $n$. It thus follows that the optimal $\alpha_{n,k}$ are satisfied
by the equations $f_{n,k}(\alpha_{n,k})=f_{n,k+1}(\alpha_{n,k})$. This affords
us the following closed form:

$\alpha_{n,k}=1/\left(1+\frac{p_{k}p_{n-k-1}}{p_{k-1}p_{n-k}}\left(\frac{n}{k}-1\right)\right)$
(3)

Figure 1: Illustration of advantage of risk-tolerant strategy over equal-spacing strategy for a 5-number game. Black curves correspond to $f_{5,k}(x)$ and blue vertical dashed lines indicate strategy boundaries. Volume of red area equals probability of winning the 5-number game given the blue decision boundaries and optimal play after. (_Top_) Equal-spacing strategy (_Bottom_) Risk-tolerant strategy. Notice entire volume filled in the bottom figure. Order |  Sequence so far |  Prob correct so far |  Win prob (RT) |  Win prob (ES)
---|---|---|---|---
1 |  |  |  |
2 |  |  |  |
3 |  130 |  |  |
4 |  |  $9.58\times 10^{-3}$ |  $1.12\times 10^{-4}$ |  $1.02\times 10^{-4}$
5 |  |  $6.81\times 10^{-3}$ |  $1.28\times 10^{-4}$ |  $1.17\times 10^{-4}$
6 |  |  $2.07\times 10^{-3}$ |  $4.61\times 10^{-5}$ |  $4.26\times 10^{-5}$
7 |  |  $3.51\times 10^{-4}$ |  $8.37\times 10^{-6}$ |  $7.74\times 10^{-6}$
8 |  |  $3.56\times 10^{-5}$ |  $8.50\times 10^{-7}$ |  $7.85\times 10^{-7}$
9 |  |  $2.17\times 10^{-6}$ |  $4.81\times 10^{-8}$ |  $4.46\times 10^{-8}$
10 |  |  $7.33\times 10^{-8}$ |  $1.37\times 10^{-9}$ |  $1.25\times 10^{-9}$
11 |  |  $1.06\times 10^{-9}$ |  $1.25\times 10^{-11}$ |  $1.13\times 10^{-11}$
12 |  573 |  |  |
13 |  |  |  |
14 |  |  |  |
15 |  |  |  |
16 |  761 |  |  |
17 |  |  |  |
18 |  |  |  |
19 |  |  |  |
20 |  |  |  |
Table 1: Illustration of a 20-number challenge where probabilities displayed
correspond to placing the next number 170 in slot $k$. Notice that the equal-
spacing strategy (i.e. deciding based on maximizing probability correct so
far) would dictate that 170 should be placed in slot 4, whereas maximizing
probability of winning using the risk-tolerant strategy suggests that 170
should be placed in slot 5. By placing 170 in slot 5 and continuing to play
risk-tolerant, the odds of winning are about 25% greater than placing 170 in
slot 4 and continuing to play with equal-spacing.

We present a variety of illustrations to elucidate the advantage of using the
risk-tolerant strategy versus the equal-spacing strategy. Let $p_{n}^{ES}$ and
$p_{n}^{RT}$ correspond to win probabilities of the $n$-number game using the
equal-spacing and risk-tolerant strategies respectively. Figure 2 demonstrates
that both $p_{n}^{ES}$ and $p_{n}^{RT}$ approximately exponentially decrease
as $n$ increases, and further the factor of advantage of risk-tolerant over
equal-spacing (i.e. $p_{n}^{RT}/p_{n}^{ES}$) is approximately linear in $n$.
For example, in a 20-number game the equal spacing strategy gives a win
probability of $p_{20}^{ES}\approx 1/9,651$ while the risk-tolerant strategy
achieves $p_{20}^{RT}\approx 1/7,980$, the improvement factor being
$p_{20}^{RT}/p_{20}^{ES}\approx 1.2095$. The advantage is more pronounced for
larger number games; in a 40-number game the win probabilities satisfy
$p_{40}^{ES}\approx 1/433,000,000$, $p_{40}^{RT}\approx 1/296,000,000$, and an
improvement factor of $p_{40}^{RT}/p_{40}^{ES}\approx 1.4617$

Figure 2: Comparison of equal-spacing strategy versus risk-tolerant strategy
(_Left_) log-scale probability of winning (_Right_) factor of improvement in
win probability of risk-tolerant over equal-spacing.

Figures 3 and 4 illustrate the decision boundaries corresponding to each
strategy, which we denote as $\alpha_{n,k}^{ES}$ and $\alpha_{n,k}^{RT}$. In
particular, we find that the most extreme differences in bin sizes between
equal-spacing and risk-tolerant strategies occur in the first and last bins,
namely the size of the end bins in the risk tolerant strategy is about 60% of
the size of an equal-spacing bin (i.e.
$\lim_{n\rightarrow\infty}n\alpha_{n,1}\approx 0.6$). Meanwhile, the second
and third risk-tolerant bins (equivalently second and third to last bins) have
volume slightly less than $1/n$ each, and the remainder of the bins
approximately evenly divide the remainder.

Figure 3: Comparison of bin boundaries for different strategies. Each row
indicates an $n$-number game where $n\in\\{2,...,20\\}$. Blue ticks indicate
boundary for risk-tolerant strategy, red ticks indicate boundary for equal
spacing strategy. Figure 4: Relative bin size (as a factor of the equal-
spacing bin size $1/n$) of the risk-tolerant strategy for $n=40$.

While the equal-spacing strategy may seem more intuitive, the risk-tolerant
strategy suggests that one should exercise caution when placing numbers at the
ends of the distribution. Interestingly, the second author has played tens of
thousands of 20-number games, and prior to any formal mathematical analysis
has developed their own instinctive strategy strongly mimicking the risk-
tolerant strategy.

##  5 Considerations for multiple games

We have just demonstrated that the risk-tolerant strategy outperforms the
equal-spacing strategy in likelihood of winning, but we are missing a key
practical component of the game. Since winning the 20-number challenge is
exceedingly rare, one would likely play until they win once, and since this is
a time-intensive endeavor, more important than maximizing the win probability
of a single game is _minimizing_ the expected time playing until the game is
won. Let $f_{n}^{\sigma}:\\{1,...,n\\}\rightarrow[0,1]$ be a conditional
probability distribution where $f_{n}^{\sigma}(k)$ is the probability that the
player using strategy $\sigma$ is eliminated at the $k^{\text{th}}$ turn
_given_ that the player does not win. Then from geometric series arguments it
follows that $\mathbb{E}_{n}(\sigma)$, the expected number of random elements
drawn using strategy $\sigma$ before winning a single $n$-number challenge is
given by:

$\mathbb{E}_{n}(\sigma)=n+\left(\frac{1}{p_{n}^{\sigma}}+1\right)\mathbb{E}(f_{n}^{\sigma})$

While increasing the probability of success through a more optimal strategy
will intuitively decrease the expected time playing the $n$-number challenge
before reaching a first victory, we must be careful that this is not offset by
an increase in $\mathbb{E}(f_{n}^{\sigma})$, or otherwise that we are not
spending too much time playing rounds that do not eventually win. We can see
in Figure 5 that while $p_{n}^{RT}>p_{n}^{ES}$, the graph of $f_{n}^{RT}$ is
shifted to the right of $f_{n}^{ES}$. In particular, we estimate
$\mathbb{E}\left(f_{20}^{ES}\right)\approx 10.8414$ and
$\mathbb{E}\left(f_{20}^{RT}\right)\approx 11.5178$, so that even though an
individual 20-number game using a risk-tolerant strategy is about 21% more
likely to win than an equal-spacing strategy, a risk-tolerant player will on
average play only 13% fewer number draws before a first win than an equal-
spacing player will. A similar calculation shows that for a 40-number
challenge, the risk-tolerant player uses on average 26% fewer number draws
before winning than the equal-spacing player.

Figure 5: Distribution of probability of turn of elimination
($f_{n}^{\sigma}$) of an $n$-number game, estimated with 100,000 iterations
(_Left_) $n=20$, (_Right_) $n=40$.

##  6 Conclusion

The $n$-number game is a probability exercise with an unintuitive, yet
provably optimal strategy. While the risk-tolerant strategy improves the odds
of winning an individual game, that advantage is tempered when attempting to
win a single game as quickly as possible since the improved strategy keeps the
player in losing games for longer. The factor of advantage of risk-tolerant
over equal-spacing strategy scales approximately linearly.

Several variants of the $n$-number game have emerged that may be intractable
to derive analytically optimal strategies for, yet are interesting to inspect
nonetheless. One may consider an $n$-number challenge where the random
variables are sampled from a non-uniform distribution. A simple strategy to
this game may entail converting the distribution to percentiles and following
the risk-tolerant strategy on the transformed variables, however since subsets
of the distribution are not similar this would require several layers of
transforms to execute. Another variant of the $n$-number game that has gained
traction recently is the $n^{2}$-number grid, where instead of ordering random
variables in an ordered list, the player places the random variables in an
$n\times n$ grid where both rows and columns must be in ascending order. Since
there are several ways to arrange the same $n^{2}$ numbers in an $n\times n$
grid such that rows and columns are sorted, in general the $n^{2}$-number grid
is easier than the $n^{2}$-number challenge. However, because a grid in
progress cannot be recursively decomposed into smaller grid games the way the
$n$-number challenge can, it is unclear how to generate an optimal strategy
beyond using a greedy metric of maximizing the probability that the remaining
numbers satisfy inequalities induced by entries on the grid (see Figure 6).

(e,1.25cm,1.25cm)—c:c:c:c:c——c:c:c:c:c— &
130 573
761

(e,1.25cm,1.25cm)—c:c:c:c:c——c:c:c:c:c— & 0.16 0.34 0.41 0.35
0.81 0.32 0.05
0.86 0.15 0 0
0.48 0.01 0 0
0.16 0 0 0

Figure 6: Example of a $5^{2}$-number grid in progress with 170 as the next
number (_Left_) First three turns of the number grid (same entries as in TAB)
(_Right_) probability that the remaining 21 numbers satisfy inequalities
induced by placing 170 in position $(i,j)$. For example, if 170 is placed in
position (3,1), then of the remaining 21 numbers, one must satisfy $x<130$,
four must satisfy $x<573$, three must satisfy $130<x<573$, three must satisfy
$170<x<761$, eight must satisfy $x>170$, and two must satisfy $x>761$. The
probability of this occurring is approximately 0.86 (not to be confused with
probability 0.86 of winning from this point).

## References

  * [1] John Herrman.  How tiktok is rewriting the world.  The New York Times, 2019.  https://www.nytimes.com/2019/03/10/style/what-is-tik-tok.html.
  * [2] similarweb.  Top websites ranking.  https://www.similarweb.com/top-websites/, Accessed: October 23, 2023.
  * [3] Doron Zeilberger.  The method of creative telescoping.  Journal of Symbolic Computing, 11(3):195–204, 1991.


# An Actor-Critic Method for Simulation-Based Optimization

Kuo Li1, Qing-Shan Jia1, and Jiaqi Yan2 1K. Li and Q.-S. Jia are with the
Center for Intelligent and Networked System (CFINS), Department of Automation,
Beijing National Research Center for Information Science and Technology
(BNRist), Tsinghua University, Beijing 100084, P.R. China li-k19,
<EMAIL_ADDRESS>Yan is with the Department of Computer
Science, Tokyo Institute of Technology, 4259-J2-54, Nagatsuta-cho, Midori-ku,
Yokohama 226-8502, Japan<EMAIL_ADDRESS>work was supported by
the NSFC under Grant 62073182 and the 111 International Collaboration Project
(BP2018006).

###### Abstract

We focus on a simulation-based optimization problem of choosing the best
design from the feasible space. Although the simulation model can be queried
with finite samples, its internal processing rule cannot be utilized in the
optimization process. We formulate the sampling process as a policy searching
problem and give a solution from the perspective of Reinforcement Learning
(RL). Concretely, Actor-Critic (AC) framework is applied, where the Actor
serves as a surrogate model to predict the performance on unknown designs,
whereas the actor encodes the sampling policy to be optimized. We design the
updating rule and propose two algorithms for the cases where the feasible
spaces are continuous and discrete respectively. Some experiments are designed
to validate the effectiveness of proposed algorithms, including two toy
examples, which intuitively explain the algorithms, and two more complex
tasks, i.e., adversarial attack task and RL task, which validate the
effectiveness in large-scale problems. The results show that the proposed
algorithms can successfully deal with these problems. Especially note that in
the RL task, our methods give a new perspective to robot control by treating
the task as a simulation model and solving it by optimizing the policy
generating process, while existing works commonly optimize the policy itself
directly.

##  I INTRODUCTION

Simulation-based optimization problems arise in various disciplines such as
computer science, economics, and so on. In general, an simulation-based
optimization problem consists of maximizing (minimizing) a score (cost)
function by choosing appropriate designs from an allowed set, and can be
typically formulated as

$\displaystyle\operatorname*{argmax}\limits_{x}{}$
$\displaystyle~{}~{}~{}f_{0}(x)$ (1) s.t. $\displaystyle f_{i}(x)\leq
0,~{}i=1,2,\cdots,m,$

where $x$ is a design from the allowed set $X$ defined by the constraints
$f_{i}(x)\leq 0,~{}i=1,2,\cdots,m$, and $f_{0}(\cdot)$ is the objective
function, which is usually computationally expensive and can be queried only
for limited times. The design $x$ can either be a scalar or vector depending
on the problem.

There are various works to deal with different cases of (1), e.g., convex
optimization deals with the case that all of $f_{i}(\cdot),~{}i=1,2,\cdots,m$
and $-f_{0}(\cdot)$ are convex ( [1]), and deep learning deals with the case
that $f_{0}(\cdot)$ is totally or partially calculated by Neural Networks
(NNs). However, in both above two cases, the operation rules are designed and
utilized during the optimization process, e.g., convex optimization utilizes
the gradient or second gradient of $f_{0}(\cdot)$, and deep learning requires
the gradient to all parameters of the NNs. Thus these methods are not suited
for simulation-based optimization, where the gradient message cannot be
obtained explicitly.

In order to address the issues brought by the “incomplete” objective
functions, researchers propose to “reconstruct” it with a controllable
surrogate model, which is usually a predictive model constructed by fitting
the existing input-output pairs. Since that a certain deviation between the
surrogate model and the true simulation model $f_{0}(\cdot)$ cannot be
avoided, related methods are usually designed to alternate between updating
the surrogate model and the sampling distribution. For example, in Bayesian
Optimization (BO, [2]), the Gaussian Process (GP, [3]) serves as the surrogate
model, and the updating process alternates between proposing the best design
with the acquisition function constructed with the surrogate model and
updating the surrogate model with the latest input-output pair. BO has the
advantage of high efficiency, but there are some technical difficulties, e.g.,
finding the design $x$ with the highest score on the acquisition function is
non-trivial, especially in high dimensional cases. The acquisition function is
a predictive model, which can only predict the performance on any
$x\in\mathcal{X}$, rather than generating the best design $x^{*}$, which is
essential in many real tasks, such as identifying the best parameters for
controlling the robots. In these cases, beyond the surrogate model, a
generative model must be equipped to identify the best design $x^{*}$.

The generative model generates a distribution over $\mathcal{X}$, which
controls the sampling process. Therefore, the sampling distribution can be
bridged to a stochastic control problem ( [4]), which adjusts the sampling
distribution basing on previous information to reach higher expected score on
the simulation model $f_{0}(\cdot)$. The generative model is commonly used in
AC-based RL, e.g., DDPG ( [5]), TD3 ( [6]) and SAC ( [7]). It is also termed
as “actor” in AC framework, which generates a distribution over the action
space (similar as the design space in simulation-based optimization) for each
system state (more details can be found in papers mentioned above).

In this work, we also consider the simulation-based optimization problem,
where the operation rule of the simulation model $f(\cdot)$ cannot be utilized
in the optimizing process. Similar as BO, a surrogate model will be used to
predict the scores on untested designs. In order to overcome the difficulty of
finding the global optimal of the acquisition function, which is usually
highly non-linear, we also introduce a generative model in our framework.
Therefore, our methods inherit the framework of AC in RL area. The framework
here is especially similar to that of SAC, which is the state-of-art RL
algorithm. The predictive model is termed as critic, which predicts
$\hat{y}\approx f_{0}(x)$ for each $x\in\mathcal{X}$, and the generative model
is termed as actor, which will be updated to reach higher expected score on
$f_{0}(\cdot)$. Each iteration starts with generating designs by the actor.
These designs will be scored by the simulation model, and the obtained input-
output pairs will be used to update the critic. The final step is updating the
actor with the latest critic.

The contributions of this paper are mainly as follows:

1) Two AC-based algorithms are designed to solve the simulation-based
optimization problem, where design space can be either continuous or discrete;

2) We take two high dimensional tasks, adversarial attack and RL, as examples
to show the application on real tasks, and the results show that the proposed
algorithms can successfully find satisfying solutions, even in the high
dimensional cases;

3) Compared with existing RL methods, e.g., Dynamic Programming (DP, [8]),
Q-learning ( [9]), DDPG, and SAC, which are motivated by optimizing the policy
itself, the proposed algorithms offer a new perspective to robot control of
reinforcing the policy through optimizing the policy generating process.
Besides, the induced methods can be deployed in an off-policy fashion and
avoid the problem of sparse and delayed reward.

##  II PRELIMINARY

Let us begin with introducing the problem of interest and the Actor-Critic
(AC) framework extensively used in Reinforcement Learning (RL) technology.

###  II-A Optimization Model

The simulation model $f(\cdot)$ takes a scalar (for one dimensional case) or
vector (for multi-dimensional case) $x$ as input, and outputs corresponding
score $y$, and we aim to find out the best design, i.e.,

$\displaystyle\operatorname*{argmax}\limits_{x\in\mathcal{X}}$
$\displaystyle~{}f(x),$ (2)

where $\mathcal{X}$ is the allowed set.

In real implementations, $f(x)$ could serve as any objective function. For
example, it can be a convex optimization problem, if $f(\cdot)$ is a concave
function as well as $\mathcal{X}$ is a convex set. $f(\cdot)$ can also be a
complex nonlinear function, e.g., Neural Networks (NNs), and the problem comes
to find a special design $x$ which receives the maximal response. For another
example, in RL tasks, the agent usually intends to find a parameterized
policy, which can gather higher accumulated rewards by interacting with the
environment. In this case, the environment is a simulation model $f(\cdot)$,
and the policy is a design $x$. Then it also comes to the standard form of
(2). In Section IV, we will demonstrate the applications in these tasks.

###  II-B Actor-Critic Framework

AC framework is widely used in RL area, which consists of two parts: actor,
which generates actions, and critic, which evaluates each action. In RL area,
both the actor and critic are typically constructed with NNs, where the actor
network encodes a mapping from state $s$ to action $a$, and the critic network
maps each $(s,a)$ pair to its predicted accumulated reward. Related work can
be found in DDPG ( [5]), TD3 ( [6]) and SAC ( [7]).

Typically, the critic is trained by minimizing the residual error of the
Bellman equation:

$\displaystyle
L(\phi)=\frac{1}{2}\mathop{\mathds{E}}\left[\left(Q^{\phi}(s,a)-\left(R(s,a)+\gamma\mathop{\mathds{E}}Q^{\phi}(s^{\prime},a^{\prime})\right)\right)^{2}\right],$
(3)

where $Q^{\phi}$ is the critic parameterized by $\phi$, $R(s,a)$ is the
instantaneous reward, $s^{\prime}$ is the succeeding state and $a^{\prime}$ is
the next action drawn from current policy. Besides, $\gamma\in[0,1)$ is the
discount factor for stabilizing the training process. $\phi$ can be updated by
minimizing $L(\phi)$.

The actor is updated by maximizing the expected accumulated reward:

$\displaystyle
J(\theta)=\mathop{\mathds{E}}\limits_{s}\left[\mathop{\mathds{E}}\limits_{a\sim\pi^{\theta}(s)}Q^{\phi}(s,a)\right],$
(4)

where $\pi^{\theta}$ is the current policy parameterized by $\theta$. Notice
that in this work, we choose the stochastic policy instead of the
deterministic one for its excellent performance of balancing the exploitation
and exploration.

##  III MAIN METHODS

Let us introduce the methodology of solving (2) with AC framework. The cases
where $\mathcal{X}$ is continuous and discrete will be separately considered.

(a) Actor for continuous design space

(b) Actor for discrete design space

(c) Critic

Figure 1: Illustration of the network structures. Fig. 1(a) and Fig. 1(b) are
the network structures of the actor for continuous and discrete design space
respectively. Fig. 1(c) is the network structure of the critic.

We use NNs to construct the actor and critic (Fig. 1). The actor takes a
random noise (or constant) as input to capture the multi-modal (or single-
modal) distributions, and the critic takes the design $x$ as input and outputs
a predicted score.

The actor and critic are updated iteratively. Each iteration starts with
generating a feasible design $x$ and getting its score $y=f(x)$ by querying
the simulation model $f(\cdot)$. The $(x,y)$ pair will be stored in a replay
buffer $\mathcal{D}$. Then a mini-batch $\mathcal{B}_{q}$ will be sampled from
$\mathcal{D}$ to update the critic by minimizing the loss function:

$L(\phi)=\frac{1}{2}\sum\limits_{(x,y)\in\mathcal{B}_{q}}\left(Q^{\phi}(x)-y\right)^{2},$
(5)

with one step of gradient descent:

$\nabla_{\phi}L(\phi)=\sum\limits_{(x,y)\in\mathcal{B}_{q}}\left(Q^{\phi}(x)-y\right)\nabla_{\phi}Q^{\phi}(x).$
(6)

Finally, the actor will be updated with the latest critic.

A stochastic actor is used to generate designs, and in each iteration, the
sampling distribution will be updated to improve the expected score. In order
to reduce the risk of falling into local optimal, we use entropy
regularization to reach the trade-off between exploitation and exploration.
The updating rule of the actor will be studied for continuous and discrete
design space respectively.

###  III-A Continuous Design Space

We firstly introduce the case where $\mathcal{X}$ is continuous. For the ease
of illustration, this paper focuses on the limited space
$\mathcal{X}=\left\\{\forall x~{}|~{}||x||_{\infty}<1\right\\}$. However, we
note that the results can be easily extended to any continuous limited space
through linear mapping.

Squashed Gaussian policy (similar as SAC) is used to generate designs, as
shown in Fig. 1(a). Firstly, the actor network outputs two vectors,
$\mu^{\theta}$ and $\sigma^{\theta}$, where $\theta$ is the parameter of the
actor network. Then $x$ is generated by

$x=g^{\theta}(\xi)=tanh(\mu^{\theta}+\sigma^{\theta}\odot\xi),$ (7)

where $\odot$ represents the element-wise multiplication. $\mu^{\theta}$,
$\sigma^{\theta}$ and $\xi$ have the same dimension with $x$, and $\xi$ is a
random noise taken from multi-dimensional standard Gaussian distribution
$\xi\sim\mathcal{N}(0,I)$. The $tanh$ function converts
$\mathcal{N}(\mu^{\theta},diag\left(\sigma^{\theta}\right))$ to another
distribution whose value range is $(-1,1)$, and its probability density
function can be calculated by

$\displaystyle\hbar(x|\theta)=$ $\displaystyle
h(\xi|\theta)\left|\det(\frac{dg^{\theta}(\xi)}{d\xi})\right|^{-1}$ (8)

at $\xi=(g^{\theta})^{(-1)}(x)$, where $h(\cdot|\theta)$ is the probability
density function of
$\mathcal{N}(\mu^{\theta},diag\left(\sigma^{\theta}\right))$, and
$(g^{\theta})^{(-1)}(\cdot)$ is the inverse function of $g^{\theta}(\cdot)$.

The sampling distribution will be updated to reach higher expected score

$J(\theta)=\mathop{\mathds{E}}\limits_{x\sim\hbar(\cdot|\theta)}Q^{\phi}(x)+\alpha
H(\hbar(\cdot|\theta)),$ (9)

where $H(\hbar(\cdot|\theta))$ is the entropy regularization for balancing
exploitation and exploration, and can be calculated by

$H(\hbar(\cdot|\theta))=\mathop{\mathds{E}}\limits_{x\sim\hbar(\cdot|\theta)}[-\log\hbar(x|\theta)].$
(10)

Therefore, $J(\theta)$ can be rewritten as

$\displaystyle J(\theta)=$
$\displaystyle\mathop{\mathds{E}}\limits_{x\sim\hbar(\cdot|\theta)}\left[Q^{\phi}(x)-\alpha\log\hbar(x|\theta)\right]$
(11) $\displaystyle=$
$\displaystyle\mathop{\mathds{E}}\limits_{\xi\sim\mathcal{N}(0,I)}\left[Q^{\phi}(g^{\theta}(\xi))-\alpha\log\hbar(g^{\theta}(\xi)|\theta)\right]$

by substituting (10) to (9), and its gradient with respect to $\theta$ can be
derived by:

$\displaystyle\nabla_{\theta}J(\theta)=$
$\displaystyle\mathop{\mathds{E}}\limits_{\xi\sim\mathcal{N}(0,I)}\left[\nabla_{g^{\theta}(\xi)}\left(Q^{\phi}(g^{\theta}(\xi))\right.\right.$
(12)
$\displaystyle\left.\left.-\alpha\log\hbar(g^{\theta}(\xi)|\theta)\right)\nabla_{\theta}g^{\theta}(\xi)\right],$

which can be estimated by a batch of samples $\mathcal{B}_{a}$ taken from
$\mathcal{N}(0,I)$:

$\displaystyle\nabla_{\theta}J(\theta)=$
$\displaystyle\frac{1}{N}\sum\limits_{\xi\in\mathcal{B}_{a}}\left[\nabla_{g^{\theta}(\xi)}\left(Q^{\phi}(g^{\theta}(\xi))\right.\right.$
(13)
$\displaystyle\left.\left.-\alpha\log\hbar(g^{\theta}(\xi)|\theta)\right)\nabla_{\theta}g^{\theta}(\xi)\right],$

where $N$ is the size of $\mathcal{B}_{a}$. Finally, $\theta$ can be updated
with (13) through one step of gradient ascent.

We conclude the optimization process in Algorithm 1. In each iteration, only a
single design will be taken to be validated with the optimization model, and
the data pair will be stored for future use. In the end, the best design in
the replay buffer will be chosen as the final result.

Algorithm 1 Optimization with continuous design space

1:for $episode=1:M$ do

2: The actor generates a design
$x=g^{\theta}(\xi)=tanh(\mu^{\theta}+\sigma^{\theta}\odot\xi),~{}\xi\sim\mathcal{N}(0,I).$
(14)

3: The simulation model gives the score $y=f(x)$.

4: Store $(x,y)$ pair to $\mathcal{D}$.

5: for $round=1:K$ do

6: Sample a mini-batch $\mathcal{B}_{q}$ from $\mathcal{D}$.

7: Update the critic with gradient estimated by (6).

8: for $round=1:K$ do

9: Sample a mini-batch $\mathcal{B}_{a}$ from $\mathcal{N}(0,I)$.

10: Update the actor with gradient estimated by (13).

11:return the best design stored in $\mathcal{D}$.

###  III-B Discrete Design Space

Here we consider the case that there are only finite feasible designs in
$\mathcal{X}$. In this case, the network structure of the actor will be
slightly modified to output a discrete distribution over the design space. As
illustrated in 1(b), the number of the output channels equals the size of
$\mathcal{X}$, and each channel corresponds to a feasible design. After
applying the softmax function, a distribution over $\mathcal{X}$ will be
obtained:

$P^{\theta}(x_{i})=\frac{\exp\left(\mu^{\theta}(x_{i})\right)}{\sum\limits_{x_{j}\in\mathcal{X}}\exp\left(\mu^{\theta}(x_{j})\right)},~{}\forall
x_{i}\in\mathcal{X},$ (15)

where by slightly abusing of notation, $\mu^{\theta}(x_{i})$ here denotes the
output of the channel corresponding to $x_{i}$. Then we update the actor by
maximizing the expected score:

$\displaystyle J(\theta)=$ $\displaystyle\mathop{\mathds{E}}\limits_{x\sim
P^{\theta}}Q^{\phi}(x)+\alpha H(P^{\theta}(\cdot))$ (16) $\displaystyle=$
$\displaystyle\sum\limits_{x\in\mathcal{X}}P^{\theta}(x)\left[Q^{\phi}(x)-\alpha\log
P^{\theta}(x)\right],$

where the entropy regularization is also introduced as in Section III-A. Then,
the gradient of $J$ with respect to $\theta$ can be calculated by:

$\displaystyle\nabla_{\theta}J(\theta)=\sum\limits_{x\in\mathcal{X}}\nabla_{\theta}P^{\theta}(x)\left[Q^{\phi}(x)-\alpha\log
P^{\theta}(x)-\alpha\right],$ (17)

and the actor can be updated with one step of stochastic gradient ascent.

Algorithm 2 Optimization with discrete design space

1:for $episode=1:M$ do

2: The actor generates a design $x\sim P^{\theta}(\cdot)$.

3: The simulation model gives the score $y=f(x)$.

4: Store $(x,y)$ pair to $\mathcal{D}$.

5: for $round=1:K$ do

6: Sample a mini-batch $\mathcal{B}_{q}$ from $\mathcal{D}$.

7: Update the critic with gradient estimated by (6).
$\nabla_{\phi}L(\phi)=\sum\limits_{(x,y)\in\mathcal{B}_{q}}\left(Q^{\phi}(x)-y\right)\nabla_{\phi}Q^{\phi}(x).$
(18)

8: for $round=1:K$ do

9: The actor generates the distribution $P^{\theta}(\cdot)$.

10: The critic predicts score for each design $Q^{\phi}(\cdot)$.

11: Update the actor with gradient estimated by (17).

12:return the best design stored in $\mathcal{D}$.

We conclude the optimization process in Algorithm 2. It is worth noting that
in each update of the actor, we attempt to maximize the objective function
(16), whose optimal solution is the energy-based policy ( [10]):

$P^{*}(x_{i})=\frac{\exp\left(\frac{Q^{\phi}(x_{i})}{\alpha}\right)}{\sum\limits_{x_{j}\in\mathcal{X}}\exp\left(\frac{Q^{\phi}(x_{j})}{\alpha}\right)},~{}~{}\forall
x_{i}\in\mathcal{X}.$ (19)

There is a simple proof of (19) by constructing following optimization
problem:

$\displaystyle\operatorname*{argmin}\limits_{P(x)}{}$
$\displaystyle-\sum\limits_{x\in\mathcal{X}}P(x)\left[Q^{\phi}(x)-\alpha\log
P(x)\right]$ (20) s.t. $\displaystyle~{}~{}~{}~{}~{}~{}~{}-P(x)<0,~{}\forall
x\in\mathcal{X}$ $\displaystyle\sum\limits_{x\in\mathcal{X}}P(x)-1=0.$

It is easy to verify that (20) is a convex problem and satisfies the Slater
condition ( [1]), which guarantees that $P^{*}(x)$ is the unique solution of
the KKT conditions:

$\displaystyle P^{*}(x)$ $\displaystyle>0,$ $\displaystyle\forall$
$\displaystyle x\in\mathcal{X}$ (21a)
$\displaystyle\sum\limits_{x\in\mathcal{X}}P^{*}(x)$ $\displaystyle=1$ (21b)
$\displaystyle\lambda^{*}(x)$ $\displaystyle\geq 0,$ $\displaystyle\forall$
$\displaystyle x\in\mathcal{X}$ (21c) $\displaystyle\lambda^{*}(x)P^{*}(x)$
$\displaystyle=0,$ $\displaystyle\forall$ $\displaystyle x\in\mathcal{X}$
(21d) $\displaystyle-Q^{\phi}(x)+\alpha\log P^{*}(x)$
$\displaystyle+\alpha-\lambda^{*}(x)+\nu^{*}$ $\displaystyle=0,$
$\displaystyle\forall$ $\displaystyle x\in\mathcal{X},$ (21e)

where $\lambda^{*}$ and $\nu^{*}$ are the optimal value of dual variables.
Combining (21a), (21c) and (21d), we can get $\lambda^{*}(x)=0,\forall
x\in\mathcal{X}$. Then, subtracting it into (21e), we get

$\log P^{*}(x)=\frac{Q^{\phi}(x)}{\alpha}+c,~{}\forall x\in\mathcal{X},$ (22)

where $c=-1-\frac{\nu^{*}}{\alpha}$. By subtracting (22) into (21b), we have

$c=-\log\sum\limits_{x\in\mathcal{X}}\exp\left(\frac{Q^{\phi}(x)}{\alpha}\right).$
(23)

Finally, according to (22) and (23), the optimal solution (19) can be
obtained. In Section IV, we will show that the learned distribution is very
close to the optimal distribution $P^{*}$.

So far, we have introduced the methods to solve the simulation-based
optimization problem 2 with AC framework, and proposed algorithms for
continuous and discrete design space respectively. In the next section, we
will design some experiments to validate them.

##  IV APPLICATIONS

In this section, we will start from a toy example, with which some intuitive
interpretation of Algorithm 1 and 2 will be given. Then we design some complex
tasks, e.g., adversarial attack and reinforcement learning, to show the
effectiveness of the proposed algorithms in large-scale systems.

Firstly, we choose the probability density function of a Gaussian Mixture
Model (GMM) as the toy example:

$\displaystyle f(x)=$ $\displaystyle
w_{1}\times\frac{1}{\sqrt{2\pi}\sigma_{1}}\exp\left(-\frac{(x-\mu_{1})^{2}}{2\sigma_{1}^{2}}\right)$
(24)
$\displaystyle+w_{2}\times\frac{1}{\sqrt{2\pi}\sigma_{2}}\exp\left(-\frac{(x-\mu_{2})^{2}}{2\sigma_{2}^{2}}\right),~{}x\in(-1,1),$

and the parameters are set as $w_{1}=0.51$, $w_{2}=0.49$, $\mu_{1}=-0.7$,
$\mu_{2}=0.7$, $\sigma_{1}=0.6$, $\sigma_{2}=0.6$.

(a) $\alpha=10^{-1}$

(b) $\alpha=10^{-2}$

(c) $\alpha=10^{-3}$

(d) $Deterministic$

Figure 2: Results of Algorithm 1. We choose $\alpha=10^{-1}$, $\alpha=10^{-2}$
and $\alpha=10^{-3}$ in the experiments, and the results are shown in Fig.
2(a), 2(b) and Fig. 2(c), respectively. In each figure, the blue line is the
ground truth of the simulation model $f(\cdot)$. The orange line is the value
predicted by the critic $Q^{\phi}(\cdot)$. The green line is the sampling
distribution $\hbar(\cdot|\theta)$ generated by the actor. The curves are
normalized to $[0,1]$ for the convenience of comparison. In order to show the
effect of the entropy regularization, we replace the stochastic actor with a
deterministic one, and the result is shown in Fig. 2(d).

We launch Algorithm 1 and the results are shown in Fig. 2. It can be found
that the sampling distribution gradually converges to the optimal area as
$\alpha$ decreases, which means by setting an appropriate $\alpha$ to controls
the strength of the entropy regularization, satisfying designs can be drawn
from the final actor. To analysis the effect of entropy regularization,
$\alpha$ is set as $10^{-1}$, $10^{-2}$, and $10^{-3}$ respectively. Compared
with smaller $\alpha$ in Fig. 2(b) and Fig. 2(c), the actor in Fig. 2(a)
generates almost an uniform distribution, which means that the entropy
regularization has a greater impact on the objective in (9). Besides, since
the samples distribute more evenly over the allowed set, the predicted scores
are close to the ground truth almost everywhere. With $\alpha$ decreasing, the
sampling distribution narrows to the optimal area (Fig. 2(b) and Fig. 2(c)),
which means that the actor can generate satisfying designs. However, the
samples concentrate in some areas, which leads to worse performance of the
critic in other areas, as illustrated in Fig. 2(c). We also test the
performance of a deterministic actor, as in DDPG ( [5]), where the actor only
generates a deterministic design, rather than a distribution over the design
space. The final design falls into the local optimal more easily, as
illustrated in Fig. 2(d). In real implementation, we usually set a larger
$\alpha$ in the initial phase, and decrease it step by step. In this way, the
sampling distribution can converge to the neighbor area of the global optimal,
while avoiding falling into local optimal.

(a) $\alpha=10^{-1}$

(b) $\alpha=10^{-2}$

(c) $\alpha=10^{-3}$

(d) $\alpha=10^{-3}$

Figure 3: Results of Algorithm 2. We choose $\alpha=10^{-1}$, $\alpha=10^{-2}$
and $\alpha=10^{-3}$ in the experiments. As in Fig. 2, the blue line is the
ground truth of the simulation model $f(\cdot)$ and the orange line is the
value predicted by the critic $Q^{\phi}(\cdot)$. We use histograms to
illustrate the sampling distribution among the discrete $\mathcal{X}$. The
blue bars denote the sampling distribution $P^{\theta}(\cdot)$ generated by
the actor and the orange bars denote the optimal distribution $P^{*}(\cdot)$.

The simulation model (24) is modified to test Algorithm 2 by converting the
design space $\mathcal{X}$ to a discrete set. The results are shown in Fig. 3.
It can be found that by decreasing the strength of entropy regularization, the
sampling distribution gradually converges to the optimal area. We also compare
the effect of the entropy regularization by setting $\alpha$ as $10^{-1}$,
$10^{-2}$, and $\alpha=10^{-3}$. When a larger $\alpha$ is chosen, e.g.,
$\alpha=10^{-1}$ in Fig. 3(a), the actor generates nearly uniform
distribution. When decreasing it, the learned distribution gradually converges
to designs with higher scores, as in Fig. 3(b) and Fig. 3(c). However, if
$\alpha$ is set too small, the actor may fall into local optimal, as
illustrated in Fig. 3(d). Therefore, in real implementation, we also set a
bigger $\alpha$ in the initial phase and decrease it step by step as in the
continuous case. Furthermore, it can be found that the distribution
$P^{\theta}(\cdot)$ generated by the actor is very close to the optimal one
$P^{*}(\cdot)$, which validates the theoretical analysis.

After the toy examples, we design more complex tasks to validate the proposed
algorithms.

The first task is abstracted from adversarial attack tasks. In these tasks, we
need to generate samples to “cheat” some NNs, which are trained to recognize
the objects in the input pictures. In this experiment, we train a classifier
to recognize digital numbers with the public data set MNIST. The classifier
$D(\cdot)$ takes an image $x^{\prime}$ as input and generates a distribution
over $Y=\\{0,1,2,\cdots,9\\}$, which denotes the confidence of classifying the
digital image to corresponding value. For the convenience of later
representation, the distribution is denoted as $D(x^{\prime})\in R^{10}$ and
$D(x^{\prime},i),i\in Y$ denotes the confidence of classifying the image
$x^{\prime}\in R^{W\times H}$ into $i$, where by slight abuse of notations,
here the $W$ and $H$ are the size of images. In the adversarial attack tasks,
we need to generate some fake images to enhance the data set and train a more
robust classifier.

(a) Learning curve of the training process. The curve is drawn by averaging
the confidences on $300$ samples generated by the actor after each episode.

(b) Image generated by the actor after $1000$ episodes.

Figure 4: Learning results.

In order to generate the fake images, we construct the following simulation
model:

$f(x)=D\left(r\left(\frac{x+1}{2}\right),i\right),$ (25)

where $x\in R^{(W\times H)}$ and $r(\cdot)$ is the resizing operator
converting the input vector to the appropriate shape. The linear mapping
$\frac{x+1}{2}$ converts the value range from $(-1,1)$ to $(0,1)$, which is
the same as the images in the date set. Then by solving the optimization
problem (2), the actor can generate fake images which can mislead the
classifier to believe it as $i$. We choose $i=1$ in our experiments. The
learning curve is illustrated in Fig. 4(a), where we can find that within
$1000$ episodes (only 1 sample is evaluated by $f(\cdot)$ in each episode),
the average confidence reaches over $0.95$. Before the $500$-th episode, the
average confidence is close to $0$, which may be caused by the low-accuracy
critic. After the $500$-th episode, the score increases gradually, which means
both the actor and critic are improved. One of the images generated by the
final actor is shown in Fig. 4(b). It is just some noise from the viewpoint of
humans, but the classifier tends to predict it as the digital number “1”.

Another type of fake images are also preferred, which remain the features of
the initial digital number, but mislead the classifier to make wrong decisions
by adding some special noise. In order to generate these images, we construct
following simulation model:

$f(x)=D\left(c\left(r\left(\frac{x+1}{2}\right)\times\delta+x_{j}\right),i\right),~{}i\neq
j$ (26)

where the operator $c$ clips the input variables to $[0,1]$, and $\delta$ is a
small constant controlling the noise level, which is set as $0.2$ in our
experiment. Besides, $x_{j}$ is an image from the initial data set, whose
label is $j$. In this case, the simulation-based optimization problem (2)
tends to find special noise by adding which the classifier will wrongly
classify the image from $j$ to $i$.

Figure 5: Result of adding special noise to the initial image. The orange
curve is the confidence that the classifier recognizes the noisy image as “6”,
and the blue curve is the confidence of recognizing it as “1”. The three
images in the lower row are the initial image mixed with random noise
generated at episode $0$, $1500$ and $3000$, respectively.

The result is shown in Fig. 5. After about $2000$ episodes, the actor can
generate noise “special” enough to mislead the classifier. We present the
noisy image at episodes $0$, $1500$, and $3000$. All of these images remain
the features of “6” in the viewpoint of humans, but the classifier will be
“cheated” to make wrong decisions.

(a) CartPole-v0

(b) Learning curve

Figure 6: CartPole-v0 environment and the training result. The CartPole-v0
environment takes $0$ or $1$ as action at each time step, which drives the
black box toward left or right respectively. At most $200$ steps will be
executed in each episode, and the early stop will be triggered if the pole
falls down. Each step will be rewarded by $+1$. Fig. 6(b) shows the learning
curves of Algorithm 1, which is drawn by averaging the “return” of $300$
independent experiments.

Finally, we apply Algorithm 1 to a more complex task, i.e., RL task. In
typical policy-based and AC-based RL algorithms, the policy is parameterized
by $\zeta$ and generates action corresponding to each state. In existing RL
methods, the parameter $\zeta$ is adjusted by Policy Gradient (PG) algorithms,
as in TRPO ( [11]), PPO( [12]), or Deterministic Policy Gradient (DPG)
algorithms, as in DDPG ( [5]) and TD3 ( [6]), to reach higher accumulated
reward. While in this work, we directly treat the task as a simulation-based
model, where $\zeta$ is a concrete design. We take the “CartPole-v0”
environment ( [13]) in Fig. 6(a) as example and deploy Algorithm 1 to it. The
result is shown in Fig. 6(b), where we can find that the policy (generated
design) is successfully reinforced to reach higher score. Although existing
methods in RL area have better performance in these RL tasks, our methods
offer a new perspective to robot control, which may be helpful to solve the
problem of sparse and delayed reward widely existing in RL problems.

We design these examples to show the application of our proposed methods, as
well as validate their performance on large-scale problems. The results will
be further concluded in the next section.

##  V CONCLUSIONS

In this work, we consider the simulation-based optimization problem of
selecting the best design with a computationally expensive evaluation model.
The sampling process is bridged to the policy optimization problem and solved
under the AC framework, where the critic serves as the surrogate model
predicting the scores of untested designs, and the actor encodes the sampling
distribution. We propose algorithms to update the actor and critic for
continuous and discrete design space respectively, and design experiments to
validate their effectiveness, as well as explain the applications. The results
show that our methods can successfully find satisfying designs and avoid
falling into the local optimal to a certain extent. We note that in the
experiment of the RL task, we offer a new perspective to robot control of
optimizing the policy generating process, rather than optimizing the policy
itself. This may be helpful to solve the problem of sparse and delayed reward
in existing works. The new perspective will also be studied in future work.
Another direction that will be researched in the future is adjusting the
hyper-parameter $\alpha$ automatically, as in [14].

## References

  * [1] S. Boyd, S. P. Boyd, and L. Vandenberghe, _Convex optimization_. Cambridge university press, 2004.
  * [2] M. Pelikan, D. E. Goldberg, E. Cantú-Paz, _et al._ , “Boa: The bayesian optimization algorithm,” in _Proceedings of the genetic and evolutionary computation conference GECCO-99_ , vol. 1. Citeseer, 1999, pp. 525–532.
  * [3] R. M. Dudley, “Sample functions of the gaussian process,” _Selected Works of RM Dudley_ , pp. 187–224, 2010.
  * [4] Y. Peng, E. K. Chong, C.-H. Chen, and M. C. Fu, “Ranking and selection as stochastic control,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 8, pp. 2359–2373, 2018.
  * [5] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [6] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1587–1596.
  * [7] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1861–1870.
  * [8] R. Bellman, “Dynamic programming,” _Science_ , vol. 153, no. 3731, pp. 34–37, 1966.
  * [9] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [10] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1352–1361.
  * [11] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International conference on machine learning_. PMLR, 2015, pp. 1889–1897.
  * [12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [13] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [14] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.


# Rotation effect on the deconfinement phase transition
in holographic QCD

Jia-Hao Wang  College of Science, China Three Gorges University, Yichang
443002, China  Sheng-Qin Feng  Corresponding author<EMAIL_ADDRESS>College of Science, China Three Gorges University, Yichang 443002, China
Center for Astronomy and Space Sciences and Institute of Modern Physics, China
Three Gorges University, Yichang 443002, China  Key Laboratory of Quark and
Lepton Physics (MOE) and Institute of Particle Physics,
Central China Normal University, Wuhan 430079, China

###### Abstract

Abstract: The impact of rotation on the deconfinement phase transition under
the EM system of the soft and the hard wall models in holographic quantum
chromodynamics is studied in this paper. The metric by cylindrical coordinates
with rotation is introduced into the system to calculate the Hawking
temperature. The first holographic study on the influence of the radius of a
homogeneous rotating system on the phase diagram is proposed. It is found that
the phase transition temperature hardly changes with the rotation angular
velocity for a small rotation radius. Only with a larger rotation radius can
the change in rotational angular velocity significantly alter the phase
transition temperature. The phase transition temperature decreases rapidly
with the increase of rotation angular velocity as the rotation radius
increases.

##  I Introduction

The phase and properties of matter become very extraordinary under rotation,
which has recently attracted a lot of interest among people. This type of
research is particularly relevant to the strong interacting matter in quantum
chromodynamics (QCD). For example, astrophysical objects composed of dense QCD
matter such as neutron stars can spin rapidly Grenier and Harding (2015); et
al. (2016). Typical noncentral nucleus-nucleus collisions can generate QCD
matter that carries a nonzero angular momentum of the order of
$10^{4}$-$10^{5}\hbar$ with local angular velocities in the range of 0.01-0.1
GeV Liang and Wang (2005); Deng and Huang (2016); Pang  _et al._ (2016) in
relativistic heavy ion collision experiments. An impressive progress Yamamoto
and Hirono (2013) has also achieved to study the rotating QCD matter by using
lattice simulations.

Exploring the effect of rotation on the phase transition of QCD matter is also
very significant in relativistic heavy ion collisions. As is well known,
noncentral nucleus- nucleus collisions can generate strong external magnetic
fields Voronyuk  _et al._ (2011); Bzdak and Skokov (2012); Deng and Huang
(2012); Zhong  _et al._ (2014); She  _et al._ (2018); Guo  _et al._ (2019),
which have interesting effects on the thermodynamics and phase diagrams of QCD
matter Cao  _et al._ (2021); Grefa  _et al._ (2022); Erlich  _et al._ (2005);
Karch  _et al._ (2006); Kelley  _et al._ (2011); Sui  _et al._ (2010). The
study of the magnetic catalytic and inverse magnetic catalytic properties of
chiral condensates by using the NJL model Xu  _et al._ (2021); Qiu and Feng
(2023); Qiu  _et al._ (2023); Zhu and Feng (2023), as well as the study of the
effect of magnetic fields on phase diagrams by using holographic methods Chen
_et al._ (2018); Deng and Feng (2022); Rodrigues  _et al._ (2018a, b, c);
McInnes (2016); Gürsoy  _et al._ (2018, 2017); Dudal and Mertens (2018) are
currently in progressing smoothly. Given the close analogy between magnetic
field and rotation, it is tempting to ask how rotation could affect phase
transitions. In this article, we will use holographic methods to study the
effect of rotation on QCD phase transition.

It is well known that the confinement phase is at a lower temperature and
density, while the deconfinement phase of QCD is at a higher temperature and
density. The exploration of the phase structure of QCD is an important and
challenging topic. How to probe the phase diagrams in the $T-\mu$ plane is a
rather hard job because the QCD coupling constant becomes very large near the
phase change region, and the traditional perturbation QCD method cannot be
used. For a long time, the lattice QCD method is token as the only credible
way to study the program. Although lattice QCD works well under zero baryon
density, there is a sign problem when considering finite baryon density.
However, the most interesting region in the QCD phase diagram is the region
with finite baryon density. The QGP generated by heavy ion collisions, as well
as compact stars in astrophysics, has a finite baryon density.

In holographic QCD, various approaches exist to study the deconfinement phase
transition. Among these, using the Hawking-Page phase transition Hawking and
Page (1983); Witten (1998) from gravitational theory to investigate the QCD
deconfinement phase transition proves to be highly effective. The primary
physical idea involves the phase transition between thermal anti-de Sitter
(AdS) at low temperature and the Schwarzschild AdS black hole at high
temperature in pure gauge theory Witten (1998), which corresponds to the
confinement-deconfinement phase transition between hadron state and QGP state
in QCD. Reference Herzog (2007) delved into the Hawking-Page phase transition
between the tAdS space and the Schwarzschild AdS black hole at zero chemical
potential. Reference Lee  _et al._ (2009); Park  _et al._ (2011); Park (2010)
examined the Hawking-Page phase transition between thermal charged (tc) AdS
space and the Reissner-Nordstr$\ddot{o}$m (RN) AdS black hole at nonzero
chemical potential.

In heavy ion collisions, in addition to the presence of a strong magnetic
field, the plasma also exhibits significant angular momentum. There have been
some studies that introduce rotation into holography Braga  _et al._ (2022);
Gaete  _et al._ (2017); Erices and Martínez (2018) and effective field theory
Sun and Huang (2022); Jiang and Liao (2016). Braga Braga  _et al._ (2022)
utilized the corresponding Hawking-Page phase transition between the thermal
AdS space and the Schwarzschild AdS black hole, and selected a rotating
cylindrical symmetric black hole model to study the confinement-deconfinement
phase transition and thermodynamics of a rotating system with zero chemical
potential. In this article, we will expand corresponding Hawking-Page phase
transition into between thermal charged anti-de Sitter (AdS) space and RNAdS
black hole, discussing the confinement-deconfinement phase transition and
thermodynamic properties of nonzero chemical potentials under rotational
conditions.

The purpose of this article is to investigate the effect of plasma rotation on
the phase transition temperature $T_{c}$. This article will use holographic
AdS/QCD models, especially hard wall Polchinski and Strassler (2002); Boschi-
Filho and Braga (2004, 2003) and soft wall Karch  _et al._ (2006); Bayona  _et
al._ (2008) models, to discuss the dependence of phase transition temperature
$T_{c}$ on rotational angular velocity, rotational radius and chemical
potential. The structure of the paper is as follows: In Sec. II, we introduce
the rotation into holographic QCD and discuss the consequent impacts on
temperature and chemical potential. Section III delves into the effects of
rotation on the deconfinement phase transition by utilizing both the hard and
soft wall models. A summary and discussion are presented in Sec. IV.

##  II The holographic system under rotational background

In this section, we will introduce some common theoretical settings in the
context of rotation. To analyze the rotating charged black hole, it is
necessary to assume that the spacetime has cylindrical symmetry Gaete  _et
al._ (2017); Erices and Martínez (2018); Lemos (1995); Lemos and Zanchin
(1996); Nadi  _et al._ (2019). Consequently, the metrics of Lorentzian
signature in the cylindrical form of both AdS black hole (bh) and thermal
charged (tc) AdS space can be uniformly reformulated as

$ds_{bh,tc}^{2}=\frac{L^{2}}{z^{2}}(-f_{bh,tc}(z)dt^{2}+d\vec{x}^{2}+l^{2}d\phi^{2}+\frac{1}{f_{bh,tc}(z)}dz^{2}),$
(1)

where $L$ is AdS radius, $\phi$ is the angular coordinate describing the
rotation, and $l$ is the radius of the rotation axis. The metric function of
Eq. (1) for the background geometry of AdS black hole is given as

$f_{bh}(z)=1-mz^{4}+q^{2}z^{6},$ (2)

where $m$ and $q$ are the black hole mass and charge, respectively.

The background geometry of thermal charged AdS space is

$f_{tc}(z)=1+q_{1}^{2}z^{6},$ (3)

where $q_{1}$ is the charge of thermal charged AdS space. Because of boundary
condition $f_{bh}(z_{h})=0$, one can obtain the mass as

$m=\frac{1}{z_{h}^{4}}+q^{2}z_{h}^{2}.$ (4)

Next, the rotation will be introduced into the QCD medium system. According to
Ref. Chen  _et al._ (2021), one obtains the rotating extension from the static
configuration through a local Lorentz boost as

$t\rightarrow\frac{1}{\sqrt{1-l^{2}\omega^{2}}}(t+l^{2}\omega\phi),$ (5)

and

$\phi\rightarrow\frac{1}{\sqrt{1-l^{2}\omega^{2}}}(\phi+\omega t),$ (6)

where $\omega$ is the angular velocity. The corresponding transformation of
the metric is given as

$ds^{2}=g_{tt}dt^{2}+g_{t\phi}dtd\phi+g_{\phi t}d\phi
dt+g_{\phi\phi}l^{2}d\phi^{2}+g_{zz}dz^{2}+g_{xx}\sum_{i=1}^{2}dx_{i}^{2},$
(7)

with

$g_{tt}=\frac{\gamma^{2}(\omega l)L^{2}}{z^{2}}(\omega^{2}l^{2}-f(z)),$ (8)
$g_{\phi\phi}=\frac{\gamma^{2}(\omega l)L^{2}}{z^{2}}(1-\omega^{2}l^{2}f(z)),$
(9) $g_{t\phi}=g_{\phi t}=\frac{\gamma^{2}(\omega l)L^{2}}{z^{2}}\omega
l^{2}(1-f(z)),$ (10) $g_{zz}=\frac{L^{2}}{z^{2}f(z)},$ (11)
$g_{xx}=\frac{L^{2}}{z^{2}},$ (12)

where $\gamma(\omega l)$ is the Lorentz factor

$\gamma(\omega l)=\frac{1}{\sqrt{1-l^{2}\omega^{2}}}.$ (13)

The Lorentz boost determines the relation between the Hawking temperature $T$
of the black hole and the horizon position. Reestablishing the canonical form
of rotation metric given by Eq. (7), as shown in Refs. Gaete  _et al._ (2017);
Lemos (1995), one yields the following metric as

$ds^{2}=\frac{L^{2}}{z^{2}}\left[-N(z)^{2}f(z)dt^{2}+\frac{dz^{2}}{f(z)}+R(z)(d\phi+P(z)dt)^{2}+\sum_{i=1}^{2}dx_{i}^{2}\right],$
(14)

with

$N(z)^{2}=\frac{(1-\omega^{2}l^{2})}{1-f(z)\omega^{2}l^{2}},$ (15)
$R(z)=\gamma^{2}l^{2}-f(z)\gamma^{2}\omega^{2}l^{4},$ (16)
$P(z)=\frac{\omega(1-f(z))}{1-f(z)\omega^{2}l^{2}},$ (17)

where $N(z)$ is the lapse function and $P(z)$ is the shift function.

Following Refs. Chen  _et al._ (2021); Brown  _et al._ (1994), one can obtain
the following expressions for the Hawking temperature of rotating black holes:

$T_{H}=-\frac{N(z_{h})f^{\prime}(z_{h})}{4\pi}.$ (18)

It should be noted that due to the Lorentz transformation of time and angular
rotation components, the expression of the chemical potential differs from
that of the static case. The quark chemical potential under rotation can be
given as Cvetic and Gubser (1999); Caldarelli  _et al._ (2000)

$\mu=A_{\mu}\chi^{\mu}|_{z=z_{h}}-A_{\mu}\chi^{\mu}|_{z=0},$ (19)

where $\chi=\partial_{t}+\Omega\partial_{\phi}$ is the Killing vector, and the
gauge potential $A_{\mu}$ under rotation is given as

$A_{\mu}=A_{t}(\gamma\delta_{\mu}^{t}+\gamma\omega l^{2}\delta_{\mu}^{\phi}).$
(20)

Substituting (20) into (19), one obtains the chemical potential as

$\mu=\mu^{\prime}\sqrt{1-\omega^{2}}.$ (21)

The Ricci tensor $R_{mn}$ and metric $g_{mn}$ are second rank tensors, which
transform under coordinate transformation of rotation to

$T_{\alpha\beta}^{\prime}=\frac{\partial x^{m}}{\partial
x^{\prime\alpha}}\frac{\partial x^{n}}{\partial x^{\prime\beta}}T_{mn}\equiv
M_{\alpha\beta}^{mn}T_{mn}.$ (22)

However, for scalar coordinate transformations, the situation is different.
For example: as one defines $\mathcal{R}$ as the curvature scalar, and
$\mathcal{R}^{\prime}$ as the curvature scalar with coordinate transformation
of rotation, this transformation relation should be

$\mathcal{R}^{\prime}=g^{mn}R_{mn}=M_{\alpha\beta}^{mn}g^{\alpha\beta}M_{mn}^{\alpha\beta}R_{\alpha\beta}=g^{\alpha\beta}R_{\alpha\beta}=\mathcal{R},$
(23)

so, we can conclude that scalar quantity is not affected by the coordinate
transformation of rotation.

##  III The phase diagram under rotation background

In this section, we will discuss holographic QCD phase transition using the
hard wall and soft wall model, respectively.

###  III.1 Hard wall model

Following Refs. Lee  _et al._ (2009); Park (2010), the Euclidean action of the
Einstein-Maxwell (EM) system describing holographic light quarks for the hard
wall model is given by:

$S=\int
d^{5}x\sqrt{G}\left[\frac{1}{2\kappa^{2}}(-\mathcal{R}+2\Lambda)+\frac{1}{4g^{2}}F_{MN}F^{MN}\right],$
(24)

where $G$ is the determinant of metric, $\kappa$ is the five-dimensional
Newton constant, $\mathcal{R}$ is the Ricci scalar, $\Lambda$ is the
cosmological constant, $g$ is the 5D gauge coupling constant, and $F_{MN}$ is
the U(1) bulk gauge field strength tensor with
$F_{MN}=\partial_{M}A_{N}-\partial_{N}A_{M}$. According to Refs. Lee  _et al._
(2009); Hawking and Ross (1995); Sin (2007), the bulk gauge field $A_{M}$ is

$\displaystyle A_{t}=\frac{\mu}{\sqrt{1-\omega^{2}l^{2}}}-\rho z^{2}\ ,\
A_{i}=A_{z}=0\ (i=1,2,3),$ (25)

where $\rho$ is the quark number density, which is related to the black hole
charge $q$

$\rho=\sqrt{\frac{3g^{2}L^{2}}{2k^{2}}}q.$ (26)

By imposing the Dirichlet boundary condition $A_{t}(z_{h})=0$ near the
boundary at horizon, one can rewrite $\rho$ as

$\rho=\frac{\mu}{z_{h}^{2}\sqrt{1-\omega^{2}l^{2}}}.$ (27)

The five-dimensional Newton constant $\kappa$, and the 5D gauge coupling
constant $g$ are given as Sin (2007),

$\frac{1}{\kappa^{2}}=\frac{N_{c}^{2}}{4\pi^{2}L^{3}},$ (28)

and

$\frac{1}{g^{2}}=\frac{N_{c}N_{f}}{4\pi^{2}L},$ (29)

where $N_{c}$ and $N_{f}$ are the number of colors and flavors. From Eq. (26)
to Eq. (29), one can obtain

$q=\sqrt{\frac{2N_{f}}{3N_{c}}}\frac{\mu}{z_{h}^{2}\sqrt{1-\omega^{2}l^{2}}}.$
(30)

The boundary condition at the IR cut-off $z=z_{IR}$ is

$A(z_{IR})=\alpha\frac{\mu}{\sqrt{1-\omega^{2}l^{2}}},$ (31)

where $\alpha$ is a parameter, which can be determined as
$\alpha=-\frac{1}{2}$ given by Ref. Lee  _et al._ (2009). The quark number
density $\rho$ is given

$\rho=\frac{3\mu}{2z_{IR}^{2}\sqrt{1-\omega^{2}l^{2}}}.$ (32)

By comparing Eq. (25) and Eq. (31), one can derive the charge of thermal
charged AdS space $q_{1}$ as

$q_{1}=\sqrt{\frac{3N_{f}}{2N_{c}}}\frac{\mu}{z_{IR}^{2}\sqrt{1-\omega^{2}l^{2}}}.$
(33)

From Eq. (24), the corresponding on-shell gravity action of RNAdS BH and tcAdS
space for the hard wall model are

$S_{BH}=\frac{L^{3}V_{3}}{\kappa^{2}}\int_{0}^{\beta}dt\int_{\epsilon}^{z_{h}}dz(\frac{4}{z^{5}}-2q^{2}z),$
(34)
$S_{tc}=\frac{L^{3}V_{3}}{\kappa^{2}}\int_{0}^{\beta^{\prime}}dt\int_{\epsilon}^{z_{IR}}dz(\frac{4}{z^{5}}-2q_{1}^{2}z),$
(35)

where $V_{3}$ is the volume of the 3D space, $\epsilon(\epsilon\to 0)$ is the
UV cut-off, $\beta$ and $\beta^{\prime}$ are the time periodicity in RNAdS BH
and tcAdS space, respectively. At $z=\epsilon$, $\beta^{\prime}$ is related to
$\beta$ as follows:

$\beta^{\prime}\sqrt{f_{t}(\epsilon)}=\beta\sqrt{f_{b}(\epsilon)}.$ (36)

The on-shell action density difference between RNAdS BH and tcAdS space is
given as:

$\Delta{\text{\LARGE$\varepsilon$}}=\frac{S_{BH}-S_{tc}}{V_{3}}=\frac{L^{3}}{\kappa^{2}}\left[\beta\int_{\epsilon}^{z_{h}}dz\left(\frac{4}{z^{5}}-2q^{2}z\right)-\beta^{\prime}\int_{\epsilon}^{z_{IR}}dz\left(\frac{4}{z^{5}}-2q_{1}^{2}z\right)\right].$
(37)

The free energy density is given as $F=-(T\ln\mathcal{Z})/V_{3}$, where
$\mathcal{Z}$ is the partition function. It should be noted that, in our
pursuit of studying the thermodynamics of this hard wall model, we utilize the
compactified imaginary time metric (Euclidean signature). Thus, the free
energy is expressed as:

$F=-\frac{T\ln\mathcal{Z}}{V_{3}}\approx\frac{TS_{E}}{V_{3}}.$ (38)

Therefore, the difference of free energy density between RNAdS BH and tcAdS
space is

$\displaystyle\Delta F=$ $\displaystyle
T\Delta{\text{\LARGE$\varepsilon$}}=\frac{3N_{c}(-1+\omega^{2}l^{2})+N_{f}z_{h}^{2}\mu^{2}}{18N_{c}^{2}\pi(1-\omega^{2}l^{2})^{\frac{3}{2}}z_{IR}^{4}z_{h}^{5}}$
(39)
$\displaystyle\times[-3N_{c}(-1+\omega^{2}l^{2})(z_{IR}^{4}-2z_{h}^{4})+N_{f}z_{IR}^{2}z_{h}^{2}(2z_{IR}^{2}-9z_{h}^{2})\mu^{2}],$

where $z_{IR}=1/(0.323GeV)$ is obtained from the lightest $\rho$ meson mass
Herzog (2007).

The dependence of $\Delta F$ on $z_{h}$ with different $\mu$ and $\omega$ for
the hard wall with $m=N_{f}/N_{c}=1$ is manifested in Fig. 1.

When $\Delta F$ is positive, the tcAdS space is stable. On the contrary, when
$\Delta F$ is negative, the RNAdS black hole is stable. According to Hawking-
Page phase transition, the tcAdS space corresponds to confinement phase, the
RNAdS black hole corresponds to deconfinement phase. Therefore, a positive
(negative) $\Delta F$ suggests that the QCD system is in the confinement
(deconfinement) phase. Thus, when the difference of free energy density equals
zero, the corresponding horizon position marks the phase transition point for
the Hawking-Page phase transition or confinement-deconfinement phase
transition.

Figure 1:  The difference of free energy density of rotational system $\Delta
F$ as a function of the horizon position $z_{h}$ for different chemical
potential for the hard wall model. (a) with $\omega$ = 0 GeV. and (b) with
$\omega$ = 0.5 GeV

From Fig. 1(a) and 1(b), one finds that at a constant angular velocity
$\omega$, the horizon position of the phase transition point becomes small
with the increase of the chemical potential $\mu$. Comparing Fig. 1(a) and
Fig. 1(b), one also finds that at a fixed $\mu$, the horizon position of the
phase transition point becomes small with the increase of the angular velocity
$\omega$. It is found that chemical potential and angular velocity have the
same effects on the horizon transition position., both of which cause the
position of the phase transition point to decrease. Let us figure out what
cause the reason in the following.

In the context of a rotating QCD system, we know that the introduction of
rotation alters various physical quantities. Refs. Zhao  _et al._ (2023); Sun
_et al._ (2023) point out that the influence of rotation on thermodynamic
quantities should have a modification of the thermodynamic relation as

$\epsilon=-P+Ts+\mu\rho+\omega J,$ (40)

where $\epsilon$, $P$, $s$, $\rho$, and $J$ are the energy density, pressure,
entropy density, quark number density, and angular momentum. In (40), it is
evident that the energy density of the system also increases with $\omega$ and
$\mu$ increase. Following the law of the black hole thermodynamics Natsuume
(2015), the energy density of black hole correlates with its other properties
as follows:

$dE\propto dM\propto dA\propto dr\propto\frac{1}{z_{h}},$ (41)

where $M$, $A$, and $r$ denote the mass, area and radius of black hole,
respectively. One can find that the horizon position decreases with the
increase of energy density. This explains why $z_{h}$ decreases with the
increase of $\omega$ and $\mu$.

In order to describe the status of a two-dimensional ideal rotating system,
two critical physical quantities must be considered: angular velocity and
radii of rotation. Therefore, our study encompasses not only the variation of
phase transitions with rotation and chemical potential but also its phase
diagram on the radius of rotation.

In Fig. 2, the phase transition temperature is shown as a function of the
angular velocity with different chemical potential for different rotational
radiuses $l$ = 0.3GeV-1, $l$ = 0.6GeV-1 and $l$ = 0.9GeV-1, respectively. One
finds that small angular velocity has less impact on transition temperature
while large angular velocity induces a quick decrease of phase transition
temperature. It is also found that the critical temperature $T_{c}$ decreases
with the increase of angular velocity but has weak dependence of chemical
potential, which is analogous to some holographic work and PNJL model of
deconfinement transition of light flavor McLerran  _et al._ (2009); Chen  _et
al._ (2020); Abuki  _et al._ (2008). There it has been found that the presence
of anisotropy of rotation causes easier dissociation of the $Q\bar{Q}$ and
that in phase transitions anisotropy takes a catalytic reduction of the
critical temperature.

Since we are discussing the rotational system of QCD medium, the phase
transition feature should rely on the finite size of the rotational system.
Due to the cylindrical symmetry, these quantities are dependent on the
rotation radius $l$. It would be interesting to probe how the phase transition
feature in a strongly interacting rotating matter depends on the radius of the
rotating system. The properties as a function of the radius of the rotating
system may be connected to experimental observations in the future. And due to
the limitation of the speed of light, it naturally causes the limitations
$\omega l\leq 1$. Comparing Fig. 2(a) ($l$ = 0.3GeV-1), Fig. 2(b) ($l$ =
0.6GeV-1) and Fig. 2 is the(c) ($l$ = 0.9GeV-1), one finds that the phase
transition temperature hardly changes with the rotation angular velocity for
small rotation radii (l = 0.3GeV-1), but as the rotation radius increases, the
phase transition temperature decreases rapidly with the increase of rotation
angular velocity.

Figure 2:  Phase diagrams $T-\omega$ with different chemical potential of
$\mu$ = 0 GeV, $\mu$ = 0.15 GeV and $\mu$ = 0.3 GeV for the hard wall model.
(a) for $l$ = 0.3 GeV-1, (b) for $l$ = 0.6 GeV-1 and (c) for $l$ = 0.9 GeV-1.

###  III.2 Soft wall model

For soft wall model Park  _et al._ (2011), the Euclidean action of the EM
system is given by:

$S=\int
d^{5}x\sqrt{G}e^{-\Phi}\left[\frac{1}{2\kappa^{2}}(-\mathcal{R}+2\Lambda)+\frac{1}{4g^{2}}F_{MN}F^{MN}\right],$
(42)

where $\Phi$ is the nondynamical scalar field with $\Phi=cz^{2}$, $c$ is the
IR energy parameter. The RNAdS BH charge $q$ and tcAdS space charge $q_{1}$
under the rotation background in the soft wall model are given as

$q=\sqrt{\frac{2N_{f}}{3N_{c}}}\frac{\mu}{z_{h}^{2}\sqrt{1-\omega^{2}}},$ (43)

and

$q_{1}=\sqrt{\frac{3N_{f}}{2N_{c}}}\frac{\mu c}{\sqrt{1-\omega^{2}}}.$ (44)

The corresponding on-shell gravity action of RNAdS BH and tcAdS space are

$S_{BH}=\frac{L^{3}V_{3}}{\kappa^{2}}\int_{0}^{\beta}dt\int_{\epsilon}^{z_{h}}dze^{-cz^{2}}(\frac{4}{z^{5}}-2q^{2}z),$
(45)
$S_{tc}=\frac{L^{3}V_{3}}{\kappa^{2}}\int_{0}^{\beta^{\prime}}dt\int_{\epsilon}^{\infty}dze^{-cz^{2}}(\frac{4}{z^{5}}-2q_{1}^{2}z).$
(46)

Contrasted with the hard wall model, the integral term is multiplied by a
quadratic exponent of $z$, and the upper limit of the integral extends from
$z_{IR}$ to infinity for the soft wall model. The free energy density
difference $\Delta F$ between RNAdS BH and tcAdS space is given as

$\displaystyle\Delta F=$ $\displaystyle
T\Delta{\text{\LARGE$\varepsilon$}}=-\frac{1}{6cN_{c}\sqrt{1-\omega^{2}l^{2}}z_{h}^{4}}e^{-cz_{h}^{2}}$
(47)
$\displaystyle\times\\{(6c^{3}N_{c}z_{h}^{4}e^{cz_{h}^{2}}(\omega^{2}l^{2}-1)Ei(-cz_{h}^{2})-3c^{2}z_{h}^{2}(3\mu^{2}N_{f}z_{h}^{2}e^{cz_{h}^{2}}+N_{c}(2-2\omega^{2}l^{2}))$
$\displaystyle+c(3N_{c}(e^{cz_{h}^{2}}-2)(\omega^{2}l^{2}-1)-2\mu^{2}N_{f}z_{h}^{2}e^{cz_{h}^{2}})+4\mu^{2}N_{f}(e^{cz_{h}^{2}}-1)\\},$

where the parameter $c$ can be matched to the mass of the lightest $\rho$
meson Herzog (2007), denoted as $\sqrt{c}$ = 0.338GeV, and $E_{i}(z)$ is the
exponential integral function, which can be expressed as

$Ei(z)=-\int_{-z}^{\infty}dt\frac{e^{-t}}{t}.$ (48)

Figure 3:  Phase diagrams $T-\omega$ with different chemical potential of
$\mu$ = 0 GeV, $\mu$ = 0.15 GeV and $\mu$ = 0.3 GeV for the soft wall model.
(a) for $l$ = 0.3 GeV-1, (b) for $l$ = 0.6 GeV-1 and (c) for $l$ = 0.9 GeV-1.

Figure 3 shows the phase diagrams $T-\omega$ with different chemical
potentials and different rotation radii for the soft wall model. It is found
that the characteristics of the phase diagram of soft wall case are generally
similar to that of hard wall case, but there are also some differences. For
example, when analyzing the dependence of phase transition temperature on
chemical potential, it is found that for the hard wall model, the larger the
chemical potential is, the smaller the phase transition temperature is.
However, for the soft wall model, the larger the chemical potential is, the
larger the phase transition temperature is.

###  III.3 The comparison of phase diagram between the hard wall and the soft
wall

The comparisons of phase diagram between the hard wall and the soft wall are
published in Fig.4. Compared the soft wall model with hard wall model, the
trend of phase transition temperature $T_{c}$ decreasing with angular velocity
$\omega$ is almost the same. However, the phase transition temperature given
by soft wall model is obviously larger than that of hard wall model under the
same conditions. As the rotation radius $l$ increases, the decreasing speed of
the phase transition with angular velocity for the soft wall model is faster
than that of the hard wall model.

Figure 4:  The phase diagram $T-\omega$ comparison between the hard wall model
and the soft wall model with chemical potential of $\mu$ = 0.2 GeV. (a) for
$l$ = 0.3 GeV-1, (b) for $l$ = 0.6 GeV-1 and (c) for $l$ = 0.9 GeV-1.

##  IV SUMMARY AND CONCLUSIONS

In this paper, we focus on the impact of rotation on the deconfinement phase
transition under the EM system of the soft and the hard wall model in
holographic QCD. The metric by cylindrical coordinates with rotation is
introduced into the system to calculate the Hawking temperature. Whether it is
a soft wall model or a hard wall model, the phase transition temperatures
decrease with the increase of rotation angular velocity, but have weak
dependence of chemical potential.

The first holographic study on the influence of the radius of a homogeneous
rotating system on the phase diagram is established in the article. As we are
discussing the rotating system of QCD medium, the phase transition
characteristics should depend on the finite size of the rotating system. Due
to the cylindrical symmetry of the rotating system, the rotation radius $l$
has become an important characteristic quantity of the rotating system.
Studying the dependence of phase transition characteristics of the strongly
interacting rotating matter on the radius of the rotating system is an
important research topic. It is found that the phase transition temperature
hardly changes with the rotation angular velocity for a small rotation radius,
but as the rotation radius increases, the phase transition temperature
decreases rapidly with the increase of rotation angular velocity.

## Acknowledgments

This work was supported by the National Natural Science Foundation of China
(Grants No. 11875178, No. 11475068, No. 11747115).

## References

## References

  * Grenier and Harding (2015) I. A. Grenier and A. K. Harding, C. R. Phys. 16, 641 (2015).
  * et al. (2016) A. L. W. et al., Rev. Mod. Phys. 88, 021001 (2016).
  * Liang and Wang (2005) Z. T. Liang and X. N. Wang, Phys. Rev. Lett. 94, 102301 (2005).
  * Deng and Huang (2016) W.-T. Deng and X.-G. Huang, Phys. Rev. C 93, 064907 (2016).
  * Pang  _et al._ (2016) L. G. Pang, H. Petersen, Q. Wang, and X. N. Wang, Phys. Rev. Lett. 117, 192301 (2016).
  * Yamamoto and Hirono (2013) A. Yamamoto and Y. Hirono, Phys. Rev. Lett. 111, 081601 (2013).
  * Voronyuk  _et al._ (2011) V. Voronyuk, V. D. Toneev, W. Cassing, E. L. Bratkovskaya, V. P. Konchakovski, and S. A. Voloshin, Phys. Rev. C 83, 054911 (2011).
  * Bzdak and Skokov (2012) A. Bzdak and V. Skokov, Phys. Lett. B 710, 171 (2012).
  * Deng and Huang (2012) W.-T. Deng and X.-G. Huang, Phys. Rev. C 85, 044907 (2012).
  * Zhong  _et al._ (2014) Y. Zhong, C.-B. Yang, X. Cai, and S.-Q. Feng, Adv. High Energy Phys. 2014, 1 (2014).
  * She  _et al._ (2018) D. She, S.-Q. Feng, Y. Zhong, and Z.-B. Yin, Eur. Phys. J. A 54, 48 (2018).
  * Guo  _et al._ (2019) Y. Guo, S. Shi, S. Feng, and J. Liao, Phys. Lett. B 798, 134929 (2019).
  * Cao  _et al._ (2021) X. Cao, S. Qiu, H. Liu, and D. Li, J. High Energy Phys. 08 , 005 (2021).
  * Grefa  _et al._ (2022) J. Grefa, M. Hippert, J. Noronha, J. Noronha-Hostler, I. Portillo, C. Ratti, and R. Rougemont, Phys. Rev. D 106, 034024 (2022).
  * Erlich  _et al._ (2005) J. Erlich, E. Katz, D. T. Son, and M. A. Stephanov, Phys. Rev. Lett. 95, 261602 (2005).
  * Karch  _et al._ (2006) A. Karch, E. Katz, D. T. Son, and M. A. Stephanov, Phys. Rev. D 74, 015005 (2006).
  * Kelley  _et al._ (2011) T. M. Kelley, S. P. Bartz, and J. I. Kapusta, Phys. Rev. D 83, 016002 (2011).
  * Sui  _et al._ (2010) Y.-Q. Sui, Y.-L. Wu, Z.-F. Xie, and Y.-B. Yang, Phys. Rev. D 81, 014024 (2010).
  * Xu  _et al._ (2021) K. Xu, J. Chao, and M. Huang, Phys. Rev. D 103, 076015 (2021).
  * Qiu and Feng (2023) Y.-W. Qiu and S.-Q. Feng, Phys. Rev. D 107, 076004 (2023).
  * Qiu  _et al._ (2023) Y.-W. Qiu, S.-Q. Feng, and X.-Q. Zhu, Phys. Rev. D 108, 116022 (2023).
  * Zhu and Feng (2023) X. Zhu and S.-Q. Feng, Phys. Rev. D 107, 016018 (2023).
  * Chen  _et al._ (2018) X. Chen, S.-Q. Feng, Y.-F. Shi, and Y. Zhong, Phys. Rev. D 97, 066015 (2018).
  * Deng and Feng (2022) J. Deng and S.-Q. Feng, Phys. Rev. D 105 (2022), 10.1103/PhysRevD.105.026015.
  * Rodrigues  _et al._ (2018a) D. M. Rodrigues, E. F. Capossoli, and H. Boschi-Filho, Phys. Lett. B 780, 37 (2018a).
  * Rodrigues  _et al._ (2018b) D. M. Rodrigues, E. F. Capossoli, and H. Boschi-Filho, Phys. Rev. D 97, 126001 (2018b).
  * Rodrigues  _et al._ (2018c) D. M. Rodrigues, D. Li, E. F. Capossoli, and H. Boschi-Filho, Phys. Rev. D 98, 106007 (2018c).
  * McInnes (2016) B. McInnes, Nucl. Phys. B 906, 40 (2016).
  * Gürsoy  _et al._ (2018) U. Gürsoy, M. Järvinen, and G. Nijs, Phys. Rev. Lett. 120, 242002 (2018).
  * Gürsoy  _et al._ (2017) U. Gürsoy, I. Iatrakis, M. Järvinen, and G. Nijs, J. High Energy Phys. 03 , 053 (2017).
  * Dudal and Mertens (2018) D. Dudal and T. G. Mertens, Phys. Rev. D 97, 054035 (2018).
  * Hawking and Page (1983) S. W. Hawking and D. N. Page, Commun. Math. Phys. 87, 577 (1983).
  * Witten (1998) E. Witten, Adv. Theor. Math. Phys. 2, 505 (1998).
  * Herzog (2007) C. P. Herzog, Phys. Rev. Lett. 98, 091601 (2007).
  * Lee  _et al._ (2009) B.-H. Lee, C. Park, and S.-J. Sin, J. High Energy Phys. 07 , 087 (2009).
  * Park  _et al._ (2011) C. Park, D.-Y. Gwak, B.-H. Lee, Y. Ko, and S. Shin, Phys. Rev. D 84, 046007 (2011).
  * Park (2010) C. Park, Phys. Rev. D 81, 045009 (2010).
  * Braga  _et al._ (2022) N. R. F. Braga, L. F. Faulhaber, and O. C. Junqueira, Phys. Rev. D 105, 106003 (2022).
  * Gaete  _et al._ (2017) M. B. Gaete, L. Guajardo, and M. Hassaïne, J. High Energy Phys. 04 , 092 (2017).
  * Erices and Martínez (2018) C. Erices and C. Martínez, Phys. Rev. D 97, 024034 (2018).
  * Sun and Huang (2022) F. Sun and A. Huang, Phys. Rev. D 106, 076007 (2022).
  * Jiang and Liao (2016) Y. Jiang and J. Liao, Phys. Rev. Lett. 117, 192302 (2016).
  * Polchinski and Strassler (2002) J. Polchinski and M. J. Strassler, Phys. Rev. Lett. 88, 031601 (2002).
  * Boschi-Filho and Braga (2004) H. Boschi-Filho and N. R. F. Braga, Eur. Phys. J. C 32, 529 (2004).
  * Boschi-Filho and Braga (2003) H. Boschi-Filho and N. R. F. Braga, J. High Energy Phys. 05 , 009 (2003).
  * Bayona  _et al._ (2008) C. A. B. Bayona, H. Boschi-Filho, N. R. F. Braga, and L. A. P. Zayas, Phys. Rev. D 77, 046002 (2008).
  * Lemos (1995) J. S. Lemos, Phys. Lett. B 353, 46 (1995).
  * Lemos and Zanchin (1996) J. P. S. Lemos and V. T. Zanchin, Phys. Rev. D 54, 3840 (1996).
  * Nadi  _et al._ (2019) H. Nadi, B. Mirza, Z. Sherkatghanad, and Z. Mirzaiyan, Nucl. Phys. B 949, 114822 (2019).
  * Chen  _et al._ (2021) X. Chen, L. Zhang, D. Li, D. Hou, and M. Huang, J. High Energy Phys. 07 , 132 (2021).
  * Brown  _et al._ (1994) J. D. Brown, J. Creighton, and R. B. Mann, Phys. Rev. D 50, 6394 (1994).
  * Cvetic and Gubser (1999) M. Cvetic and S. S. Gubser, J. High Energy Phys. 04 , 024 (1999).
  * Caldarelli  _et al._ (2000) M. M. Caldarelli, G. Cognola, and D. Klemm, Class. Quantum Grav. 17, 399 (2000).
  * Hawking and Ross (1995) S. W. Hawking and S. F. Ross, Phys. Rev. D 52, 5865 (1995).
  * Sin (2007) S.-J. Sin, J. High Energy Phys. 10 , 078 (2007).
  * Zhao  _et al._ (2023) Y.-Q. Zhao, S. He, D. Hou, L. Li, and Z. Li, J. High Energy Phys. 04 2023, 115 (2023).
  * Sun  _et al._ (2023) F. Sun, S. Li, R. Wen, A. Huang, and W. Xie, arXiv e-prints , arXiv:2310.18942 (2023).
  * Natsuume (2015) M. Natsuume, Lect. Notes Phys. (2015), 10.1007/978-4-431-55441-7.
  * McLerran  _et al._ (2009) L. McLerran, K. Redlich, and C. Sasaki, Nucl. Phys. A 824, 86 (2009).
  * Chen  _et al._ (2020) X. Chen, D. Li, D. Hou, and M. Huang, J. High Energy Phys. 03 2020, 073 (2020).
  * Abuki  _et al._ (2008) H. Abuki, R. Anglani, R. Gatto, G. Nardulli, and M. Ruggieri, Phys. Rev. D 78, 034034 (2008).