# [DO NOT WORK ON THIS FILE. THIS FILE just SHOWS changes as Feb 17th 5PM made
by Hongbo]

Hongbo Guo, Ruben Naeff, Alex Nikulkov, Zheqing Zhu  Meta AIUnited States
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Bandit learning algorithms have been an increasingly popular design choice for
recommender systems. Despite the strong interest in bandit learning from the
community, there remains multiple bottlenecks that prevent many bandit
learning approaches from productionalization. Two of the most important
bottlenecks are scaling to multi-task and A/B testing. Classic bandit
algorithms, especially those leveraging contextual information, often requires
reward for uncertainty estimation, which hinders their adoptions in multi-task
recommender systems. Moreover, different from supervised learning algorithms,
bandit learning algorithms emphasize greatly on the data collection process
through their explorative nature. Such explorative behavior induces unfair
evaluation for bandit learning agents in a classic A/B test setting. In this
work, we present a novel design of production bandit learning life-cycle for
recommender systems, along with a novel set of metrics to measure their
efficiency in user exploration. We show through large-scale production
recommender system experiments and in-depth analysis that our bandit agent
design improves personalization for the production recommender system and our
experiment design fairly evaluates the performance of bandit learning
algorithms.

Recommendation System, Ranking, Multi-Armed Bandit, Upper Confidence
Bound(UCB), Bayesian Optimization, Reinforcement Learning, Video
Recommendation, Short-form Videos, Exploration Efficiency Measurement,
Feedback Loop Problem

††copyright: none

##  1\. Introduction

In the recent years, the short-form video has increased in popularity on
platforms such as on Tiktok, Youtube and Instagram. As a result, improving the
recommender systems has become critical for picking up the most relevant
videos for users. A user’s interactions to a recommended video are
comprehensive, usually includes click, sharing, comments, likes or dislikes,
video consumption time, etc. All of these objectives reflect the real user
interest, and optimizing these objectives in a multi-task machine learning
model (MTML) which takes all aspects of responses into consideration is one of
the main-stream modeling choice for recommender systems～(covington2016deep;
pi2020search; chen2019co; zhao2011integrating; gomez2015netflix).

Beyond supervised learning methods mentioned above, multi-armed bandits (MAB)
applied into modern recommender systems (li2010contextual; zhou2020neural;
xu2020neural; tang2015personalized; mcinerney2018explore; mehrotra2020bandit;
elena2021survey; silva2022multi; barraza2020introduction;
mcinerney2018explore; mehrotra2020bandit) have been drawing attention, e.g.,
reported by Spotify (mcinerney2018explore; mehrotra2020bandit) and by Yahoo
(li2010contextual). MAB is a decision-making problem where an agent tries to
maximize its expected cumulative reward while repeatedly interacting with the
environment (auer2002finite; audibert2009exploration). In the scenario of
video recommendation, we adaptively choose a video as bandit arm, and then
user interaction is logged as a reward associated with this video (arm). The
model is expected to learn the best trade-off between recommending the best
video based on historical rewards (exploitation) and trying some new videos
which user might like (exploration). There are various type of such MAB
algorithms that can be applied, e.g., contextual free algorithms such as UCB
(auer2002finite; nguyen2019recommendation; lattimore2020bandit), linear
contextual algorithms such as LinUCB (li2010contextual; chu2011contextual),
non-linear contextual algorithms (zhou2020neural; zhu2022scalable) that
utilize a deep neural network rather than a simpler linear model. Some special
types also have been studied lately, e.g., adjusting bandit algorithms to fit
the federated scenario (li2022asynchronous), or exploiting the knowledge graph
to improve classical bandits algorithms (gan2022knowledge) .

Further, in a very large scale occasion where the number of arms (either items
or users) are super huge, these bandit algorithms may lead to severe scaling
issues. Pooling arms together by similarity helps to relieve this issue
(pandey2007multi; shams2021cluster; gentile2017context; gentile2014online;
nguyen2014dynamic; cesa2013gang; nguyen2019recommendation). For example ,
literature (pandey2007multi) explores the relations of different arms and take
advantages of the similarity between arms in the bandit algorithm. While in
(shams2021cluster; gentile2017context), the users are clustered into groups by
some predefined similarity so that within each group the users tend to react
similarly to a recommendation. Since each cluster is treated as one identity,
the population gets much smaller and the intra-cluster variability can be
thought of as adding noise. Yang et al. (yang2020laplacian) further studies a
social graph scenario where each social graph node represents an bandit arm,
and the edges identify the affinity between arms. The limitations of applying
clustering approach onto such a bandit graph is two folds according to
(yang2020laplacian) : it can be very expensive for large-scale graphs ; it
ignores the subtle difference between identities within the same cluster and a
result can harm granularity. The literature (nguyen2019recommendation) focuses
on cold start problem. A new identity can get a proxy UCB score from the
history of other similar arms. In our work, similar videos are clustered into
the same topic, and the arm space is reduced from millions to hundreds.

Despite the advancements in bandit learning for recommender systems, two
critical issues arise as practitioners attempt to bring these algorithms to
production : unable to scale to MTML and unfair measurement in A/B tests.
Scaling to MTML type of recommender systems is particularly difficult for
classic bandit learning algorithms. Bandit learning algorithms, especially
those leveraging contextual information or posterior sampling, requires reward
as part of the uncertainty estimation process. Since in MTML, multiple rewards
are optimized simultaneously, algorithms that leverage reward for uncertainty
estimation are unable to effectively quantify the joint uncertainty of
multiple optimization targets.

Moreover, although there has been numerous studies in applying bandit
algorithms to recommender systems, it remains purely on an online
reinforcement learning setting where there is no hypothesis testing required.
In production recommender system, however, A/B or hypothesis testing is the
key in measuring the effectiveness of any recommender system algorithms.
Different from classic supervised learning based algorithms, bandit learning
algorithms emphasize greatly on the data collection process through its
explorative nature. Such explorative behavior induces unfair evaluation for
bandit learning agents in classic A/B test setting. A reasonable experiment
requests very careful user splitting in the A/B test, otherwise overlapping
may violate the representativeness of the arms according to (geng2020online).
Also, online experiment is hard to conduct, and some work
(maksai2015predicting) adopted offline evaluation as a simulation of the
algorithm’s online performance by utilizing the log of interaction history.

In this paper, we present a novel full lifecycle of production bandit learning
design for recommender systems, through a combination of optimism based
exploration strategy and an A/B test design for fair evaluation. The rest of
the paper is presented as follows. In section 2, we explain in detail the
system of our short-form video recommender system, the problem to be
addressed, and the bandit learning life-cycle. In section 3, we conduct a
special A/B test for more accurate measurement, and introduce a framework of
exploration efficiency metrics that illustrate how our approach impacts the
system’s dynamics. Section 4 concludes the paper and briefly covers our future
work.

##  2\. Problem, Solution and System Set Up

###  2.1. Formulation of the problem

We frame recommender system (RS) design as a contextual non-parameterized
multi-armed bandit (MAB) problem where an agent interacts with the RS
environment by making an observation at time step $t$, takes an action and
receives a reward at time $t+1$. More formally, we define the RS MAB problem
as the following:

Action Space $\mathcal{A}$:  Each bandit arm is an action. In our application,
videos are clustered by utilizing some engineered features and each cluster
corresponds to one topic. The action is to choose one of the topics, i.e.,
bandit arm is the topic (not the video). All videos of the same topic are
considered as samples from the arm. The size of action space is
$\left|\mathcal{A}\right|=K$, representing the $K$ different topics and $K$ is
limited to a few hundreds. This clustering helps to reduce the action space
(shams2021cluster; gentile2017context; pandey2007multi). Also it implicitly
utilizes contextual information and video features so as to boost the bandit
algorithm’s performance.

Observation Space $\mathcal{S}$ / Context: There are two attributes in the
observation space for recommender system, user and available actions. Although
we do not leverage parameterization for modeling uncertainty, we adopt
different uncertainty calculation for different users. Also in production
recommender systems, not all recommendations in the action space is available
at time step $t$. More formally, we define $S_{t}=(C_{t},\mathcal{A}_{t})$,
where $C_{t}$ is the user and $\mathcal{A}_{t}\in\mathcal{A}$ is the subset of
available actions. Given the definitions above, at round $t+1$, the agent
selects a best action $a_{t+1}$ from the action space
$a_{t+1}\in\mathcal{A}_{t}$.

Reward Space: The reward $r_{a}(T,C_{t-1})$ is the feedback given by the
environment once action $a$ is taken for task $T$ and user $C_{t-1}$. Similar
to many other recommendation systems such as Netflix or Spotify
(gomez2015netflix; mcinerney2018explore; mehrotra2020bandit), the tasks of our
recommender system include users play, comment, share or like.

The RS is a complicated multi-stages funnel system, which consists of source
generating, early ranking, late ranking associated, and then combination of
the multiple outputs from its predecessor. This is depicted in Fig. 1. In this
work, we focus on the last ranking stage of the recommender system and assume
that the amount of information sharing between different stages is minimal.

###  2.2. UCB applied to MTML System

In this work, we adopt the optimism facing uncertainty principle and apply
Upper Confidence Bound (UCB) approach for bandit learning and execution. The
UCB formula adopted in our experiment consists of two items, the expected
reward of an action applied to a context and the exploration bonus:

(1)
$U_{t,a}(C_{t-1})=\sum_{T\in\mathcal{T}}\alpha_{T}\hat{r}_{a}(T,C_{t-1})+\gamma\sqrt{\frac{\ln
N_{t}(C_{t-1})}{N_{t,a}(C_{t-1})}}.$

Here $a$ represents the arm or topic and $\mathcal{T}$ is the set of tasks.
$N_{t}(C_{t-1})$ represents the total number of times the recommender system
interacts with the user $C_{t-1}$ and $N_{t,a}(C_{t-1})$ represents the number
of times the recommender system has recommended topic $a$ to the user
$C_{t-1}$. $\gamma$ is a hyperparameter to adjust the aggressiveness for
exploration and $\alpha_{T}$ is the importance of task $T$.

The goal of the formula above is to induce a more explorative behavior when
uncertainty is high. For example, if we tried 100 times on football topic and
got 80 clicks (assuming a single task). We want to keep recommending football
videos to the user so as to exploit. On the other hand, on another topic (arm)
such as the romantic videos, we recommended 3 videos and user clicked 1 video.
Though this 33% chance of success is not favorable, we only tried 3 times
after all. The romantic genre might actually attract this user, but we do not
know yet due to limited trials. Thus, it may be worth taking some risk and try
more.

The unique advantage of leveraging this simple approach, UCB, is its immediate
extension to MTML without many modifications needed. The contextual non-
parametric version of UCB offers an elegant estimation of the confidence set
for all the tasks and hence simultaneously explores and learns to personalize
for multiple different tasks such as play, comment, like and share.

Figure 1. Design of Bandit Learning for the Ranking Stage of Recommender
Systems

###  2.3. Measure Bandit Learning in A/B Test

Since UCB is an online bandit learning algorithm, it introduces natural
complexity when trying to apply hypothesis testing methods, such as A/B
testing, to measure the effectiveness of bandit learning algorithms. Suppose
in the test group, the bandit learning agent sacrifices its immediate
performance to explore and learn new information from the user and the data is
then shared with both itself and the control group’s agent for model posterior
update. This puts the bandit learning agent at great disadvantage given that
both groups’ agents can learn from the same valuable information but the
bandit learning agent has to sacrifice its performance to retrieve such
information. Although one might argue that the information retrieved by the
bandit learning agent is for users that do not exist in the control group,
generalization capabilities of modern deep learning techniques can extract a
substantial amount of information from these valuable information from the
test group with substantial quantities of data.

To address the issue we identify above, we propose a new A/B testing mechanism
to bound the performance of bandit learning algorithms when measuring through
hypothesis testing. We divide our users into three groups: production, control
and test group. Control and test groups only consist of a small portion of all
users, while production group represents the majority of users. In our A/B
test, we focus on control and test groups. Instead of a single phase A/B test
with all data shared for model refreshing, we will set up the experiments for
two phases.

In Phase I, we test the effect of UCB without dedicated posterior update in
each treatment group to measure the lower bound of bandit learning capability.
In this phase, the UCB score calculated in Equation 1 is applied in the test
group, but not the control group. All groups share the same $\hat{r}_{a}(T,C)$
trained on all data across production, control and test groups for reward mean
estimation. Given that all groups share the same model, we expect the test
group to personalize better first and slowly reduce its performance
improvement. With substantial amount of exploration data collected from the
test group, we expect the model to better personalize for the control group
over the long term due to deep learning models’ generalization capabilities.

In Phase II, we enable data split between test and control group and train
dedicated models only with data from their corresponding groups to perform a
proper posterior update for both groups without data leakage. This phase aims
to measure the upper bound of bandit learning performance since it puts the
control group’s model in a slight disadvantage as the model is trained with
substantially less data. Although the test group also suffers from the same
setup, its agent’s explorative nature helps it retrieve more informative data
and hence less data is naturally needed to achieve better performance. In this
phase, we expect the performance of the test group to improve on top of phase
I’s result with better exploration strategy. Phase I and Phase II together
provides a rigorous upper and lower bound of the bandit learning performance.
For more details of how Phase I and II works, please refer to Figure 2.

Figure 2. Bandit Learning A/B Test Phases

##  3\. Analysis

###  3.1. Set Up

Our experiment ran in two phases lasting for 3 and 2 weeks, respectively. In
Phase I, we applied UCB to the test group, retraining our ranking models with
the full dataset collected across production, control and test groups. All
groups are exposed to this same model. In Phase II, we retrained the model on
each group separately. In both phases, we trained the model daily. Across both
phases and both treatment groups, they share the expected reward MTML model
architecture with the same training computational resources.

When trained separately in Phase II, our treatment groups are large enough to
allow the MTML models to learn from similar users and items and generalize
across the entire group of users. Note, though, that upstream sourcing and
ranking stages are still trained on the full production data. In this
experiment, we assume that upstream models are not significantly impacted by
the introduction of UCB to the test group given the small treatment group
size.

###  3.2. Metrics

To measure the success of our approach, we look at the number of _plays_ of
short-form videos. To measure the quality of those plays, we also look their
_loop rate_ and _skip rate_ , i.e., if users watch a video repetitively, for
less than a few seconds. We consider these as our MTML tasks.

In addition, to evaluate whether our policy follows a near-optimal posterior
belief, we develop a suite of _exploration efficiency_ metrics. Using a coarse
taxonomy of 30 topics, we create posterior distributions for
$p(\text{relevant})$ for each ¡user, topic¿ pair, based on past
recommendations, where we mark the completed videos as relevant. Note that
this includes never-seen topics.111We use beta distributions with a light
prior based on topic’s and user’s averages. By sampling from these posteriors,
we can compute the probability that a topic would be the most relevant, i.e.,
we create a probability distribution $P$ over all topics. Ideally, our policy
follows this belief, so we compare this with our recommended topic
distribution $Q$.222We compute $P$ with data before today, while $Q$ is
computed with today’s data. For these metrics, we assume independence between
topics without representation learning, which is a simplification.

Now we can define the _exploration inefficiency (EI)_ as the KL divergence
between $P$ and $Q$, _topic diversity (TD)_ as the number of recommended
topics,333A more natural metric would be the entropy of the topic
distribution, which was unfortunately not available at the time. _interest
uncertainty (IU)_ as the entropy of $P$, and _topic excellence (TE)_ as the
probability that any given recommendation happens to be on the most relevant
topic.

(2) $EI=D_{\text{KL}}(P\parallel Q)=\sum_{i}p_{i}\log\frac{p_{i}}{q_{i}}$

(3) $TD=|Q|=\text{\\# topics}$

(4) $IU=H(P)=-\sum_{i}p_{i}\log p_{i}$

(5) $TE=P\cdot Q=\sum_{i}p_{i}q_{i}$

We average these user-level metrics across all users. For topic excellence,
the average is weighted by the number of plays. In the KL divergence, we
impute missing topics with a small value, which skews the metric worse for
light users.

Our experiment aims for a lower exploration inefficiency and a higher topic
excellence.

###  3.3. Findings

Figure 3. Effect on play count (test/control-1)

Comparing test with control, we find an increase bounded around +0.5% plays of
short-form videos (Fig. 3). We make three observations: plays go up in phase
I, yet after a while these gains start to diminish; then in phase II we see
again an increase. We will discuss some interpretations later on.

In our control group (Fig. 4), we see the effects of a classic _feedback loop
problem_ : for more active users we are exploring less efficiently, and while
we get a clearer picture of which topics have potential, we do not seem to be
exploring them. The shaded bell curve is the distribution of total plays per
user.

(a) exploration inefficiency

(b) diversity (# topics)

(c) interest uncertainty

(d) topic excellence

Figure 4. Exploration efficiency by user activity (control)

In Fig. 5, we see that our experiment indeed decreases our exploration
inefficiency, which stays consistently low throughout Phase I and II. As a
result, diversity increases, as uncertain topics are promoted. Yet, as
certainty grows, these promotions get weaker. In Phase II, when our model
learns, diversity grows again, as new interests are recommended. Throughout,
we get consistently better in discriminating between a user’s interests. Topic
excellence follows the same shape as play count and diversity: an initial jump
as the potential of our unexplored topics is high (we’re optimistic), steadily
decreasing as we face reality, and, in Phase II, increasing when we recommend
our newly discovered and superior topics again. To recap, exploration
inefficiency dropped, and topic excellence grew, as we had intended.

(a) exploration inefficiency

(b) diversity (# topics)

(c) interest uncertainty

(d) topic excellence

Figure 5. Effect on exploration efficiency (test/control-1)

In Fig. 6, we see the effect of exploration on loops and skips of our
recommended videos. The initial bump might point at novelty effects, as new
topics are boosted. Then rates reverse, suggesting that users have the
opportunity to explore new interests by swiping through some of the
suggestions. In Phase II, when the model learns from the newly-explored
topics, users seem to have found their interests: loop rates are higher than
in control and skip rates end up statistically tied to control.

(a) loop rate

(b) skip rate

Figure 6. Effect on video engagement (test/control-1)

To illustrate the validity of our exploration efficiency metrics, we also
analyzed a collection of hundreds of other experiments in our recommendation
funnel. In Fig. 7, we see that various _UCB_ experiments (red, Phase I only)
grow topic excellence most, i.e., showing topics that are more likely to be
the user’s most-rewarding ones. Alternatively, the _novelty_ experiments
(blue) promote not-yet seen topics, which also increase the topic excellence,
yet not as much as UCB. (The orange _gateway_ experiment is out of scope of
this paper.) The grey dots are experiments that are not necessarily exploring
new topics. Experiments with effects that were not stat. sig. are excluded
from this chart. Note that virtually all our experiments that meant to explore
user interests (i.e., UCB and novelty boosting) increase both topic excellence
and play count.

Figure 7. Effect on topic excellence (x) and play count (y) (test/control-1)

##  4\. Conclusion and future work

In this work, we formulate our short-form video recommender system as a
contextual non-parametric multi-armed bandit problem. We introduce a novel
bandit learning lifecycle for recommender systems by proposing an elegant and
effective way of introducing optimism based bandit learning to multi-task
machine learning system and a new A/B test design for fair measurement of its
performance. We show empirically that the optimism based bandit learning
approach significantly improves personalization for multiple tasks and our
newly designed A/B testing framework fairly evaluates bandit learning
performance.

Beyond the outcomes of this paper, we see substantial potential of bandit
learning algorithms for production recommender systems. One line of work that
could potentially substantially further improve the performance of bandit
learning in multi-task machine learning recommender system is to leverage
epistemic neural recommendations (zhu2022scalable) to replace each task
prediction with a posterior sample. Together with the newly designed A/B
testing framework, we could potentially build and evaluate a sophisticated
deep-learning bandit based recommender system at scale.


# Synthesis of separation processes with reinforcement learning

Stephan C.P.A. van Kalmthout
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>

&Laurence I. Midgley
Instadeep
London, United Kingdom
<EMAIL_ADDRESS>
&Meik B. Franke
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>

###### Abstract

This paper shows the implementation of reinforcement learning (RL) in
commercial flowsheet simulator software (Aspen Plus V12) for designing and
optimising a distillation sequence.111Code and thesis are available at
https://github.com/lollcat/Aspen-RL The aim of the SAC agent was to separate a
hydrocarbon mixture in its individual components by utilising distillation.
While doing so it tries to maximise the profit produced by the distillation
sequence. All actions of the agent were set by the SAC agent in Python and
communicated in Aspen Plus via an API. Here the distillation column was
simulated by use of the build-in RADFRAC column. With this a connection was
established for data transfer between Python and Aspen and the agent succeeded
to show learning behaviour, while increasing profit. Although results were
generated, the use of Aspen was slow (190 hours) and Aspen was found
unsuitable for parallelisation. This makes that Aspen is incompatible for
solving RL problems.

_K_ eywords deep learning $\cdot$ process synthesis $\cdot$ reinforcement
learning $\cdot$ distillation $\cdot$ Aspen Plus $\cdot$ soft actor-critic

##  1 Introduction

Reinforcement learning is a sub-field of machine learning, in which an agent
aims to learn an optimal policy which maximises its expected reward. The
policy is improved by setting actions inside the environment and updating the
agent based on these experiences. Previously RL has shown to be capable of
outperforming humans in Chess and Go [1], but also shown success in process
control [2, 3]. Recent research has shown the applicability of RL in process
synthesis [4, 5, 6, 7, 8]. Reinforcement learning shows potential in handling
open-end problems. Where superstructures have their optimal solution embedded
in the user-defined superstructure, allows the reinforcement learning for
exploration outside this pre-defined superstructure.

##  2 Reinforcement learning

A key characteristic of an RL problem is that it is defined as a Markov
Decision Process (MDP). This implies that the choice of the current action
only depends on the current state, see figure 1. An MDP consists of 2
entities, an agent and environment. The agent observes the state of the
environment, $S_{t}$, and sets an action accordingly, $A_{t}$. This action is
selected from the possible actions defined by policy $\pi$. Due to the action,
the environment transitions to its new state, $s_{t+1}$. With this new created
state a reward is gained, $R_{t+1}$. This sequence continues until the
terminal state is reached, which can be user defined. The agent will set
actions such that it maximises its total expected reward,
$\max_{{\pi}}\mathbb{E_{\tau\sim\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}R\right]$.

Figure 1: Markov Decision Process schematic [9]

##  3 Soft Actor Critic

Actor critic agents have been created to benefit from both policy optimisation
and Q-learning methods. The key characteristic of an actor-critic algorithm is
that the agent is composed of 2 “sub-agents”, an actor and a critic, figure 2.
An actor is learning a policy, $\pi$, which performs actions maximising the
future expected reward.

Figure 2: Actor Critic architecture [9]

To maximise exploration by the agent a soft actor critic (SAC) algorithm was
implemented. With SAC, the RL objective is modified such that is tries to
maximise the expected return and entropy of the agent,
$\max_{{\pi}}\mathbb{E_{\tau\sim\pi}}\left[\sum_{t=0}^{\infty}\gamma^{t}(R+\alpha
H(P))\right]$. The temperature parameter $\alpha$ determines the contribution
of H in the agent’s objective where H is the entropy which is a measure of how
random the actions of the agent are. Both the actor and critic are represented
by a feed-forward neural network and are parameterised by $\phi$ and $\theta$
respectively.

The network of the actor takes in the state parameters and outputs a mean,
$\mu_{\phi}$, and standard deviation, $\sigma_{\phi}$, over the action-space
of each action. Parameters of the network are updated with the aim to minimise
the KL-divergence, equation 1. $Z^{\pi_{old}}(s_{t})$ is a normalisation
factor [10].

$\pi_{new}=\arg\min_{\pi^{\prime}\in\Pi}D_{KL}\left(\pi^{\prime}(\cdot|s_{t})||\frac{\exp(\frac{1}{\alpha}Q^{\pi_{old}}(s_{t},\cdot))}{Z^{\pi_{old}}(s_{t})}\right)$
(1)

Minimising the KL-divergence is done by updating the network parameters via
stochastic gradient decent, equation 2 [9].

$J_{\pi}(\phi)=\mathbb{E}_{s_{t}\sim
D,a_{t}\sim\pi_{\phi}}\left[\alpha\log(\pi_{\phi}(a_{t}|s_{t}))-Q_{\theta}(s_{t},a_{t})-Z^{\pi_{old}}(s_{t})\right]$
(2)

Estimating the gradient with respect to $\phi$ is not possible, since the
expectation depends on $\phi$. Therefore the reparameterization trick is
applied, such that
$a_{t}=f_{\phi}(s_{t},\epsilon_{t})=\tanh(\mu_{\phi}+\sigma_{\phi}\odot\epsilon)$,
where $\epsilon$ is a noise vector sampled from a normal distribution,
$\epsilon\sim\mathcal{N}(0,1)$ equation 3.

$J_{\pi}(\phi)=\mathbb{E}_{s_{t}\sim
D,\epsilon\sim\mathcal{N}}\left[\alpha\log(\pi_{\phi}(f_{\phi}(\epsilon_{t},s_{t})|s_{t}))-Q_{\theta}(s_{t},f_{\phi}(\epsilon_{t},s_{t}))\right]$
(3)

$Z^{\pi_{old}}(s_{t})$ is neglected, since it is independent of $\phi$ and is
regarded a constant when estimating the gradient with respect to $\phi$. The
critics’ aim is to learn an optimal soft Q function, equation 4, by minimising
the mean squared error, equation 5.

$Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[r+\gamma(Q_{\theta}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}|s_{t+1}))\right]$
(4)

$Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[\frac{1}{2}(Q_{\theta}(s_{t},a_{t})-(r+\gamma\mathbb{E}_{s_{t+1}\sim
p}[Q_{\bar{\theta}}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}|s_{t+1})]))^{2}\right]$
(5)

Updates of the critic network is performed by applying stochastic gradient
descent following equation 6.

$\widehat{\nabla_{\theta}}J_{Q}(\theta)=\nabla_{\theta}Q_{\theta}\left(Q_{\theta}(s_{t},a_{t})-(r+\gamma(Q_{\overline{\theta}}(s_{t+1},a_{t+1})-\alpha\log(\pi_{\phi}(a_{t+1}|s_{t+1}))))\right)$
(6)

$Q_{\overline{\theta}}$ in equation 6 is the Q value of a target Q network.
This target network is introduced to improve stability of the critic and its
parameters are updated by Polyak averaging,
${\overline{\theta}=\tau\theta+(1-\tau)\overline{\theta}}$. $\tau$ is the
target smoothing coefficient, which makes for soft target parameter updates.

To determine the relative importance between the reward and entropy the
temperature $\alpha$ is introduced. This parameter determines the
stochasticity of the policy and is updated via a multi-constrained objective
given in equation 7, where $\overline{H}$ is a set target entropy.

$\left\\{\begin{array}[]{@{}l@{}}\pi^{*}=\arg\max\sum_{t}\mathbb{E}_{(s_{t},a_{t})\sim\pi}[r(s_{t},a_{t})]\\\
s.t.\mathbb{E}_{(s_{t},a_{t})\sim\pi}[-\log\pi(a_{t}|s_{t})]\geq\overline{H},\forall
t\end{array}\right.$ (7)

The target entropy was fixed, whilst $\alpha$ was automatically tuned via the
following objective function 8.

$J(\alpha)=\mathbb{E}_{a_{t}\sim\pi_{t}}\left[-\alpha\log\pi_{t}(a_{t}|s_{t})-\alpha\overline{H}\right]$
(8)

##  4 Case description

For this research, a hydrocarbon mixture was considered for the feed stream
with the goal to separate it in individual components with a minimum purity of
95 mol%. The feed was processed at 12.400 kmol/h with the composition stated
in appendix 2. The agents’ reward was defined as

$r(s_{t},a_{t})=revenue_{top}+revenue_{bottom}-TAC-penalty$ (9)

where revenue was only generated when the purity specification was 95 mol%.
The Total Annualized Cost (TAC) was based on the total capital expendatures
(CAPEX) and operational expedatures (OPEX) of the distillation sequence.
Appendix D shows an overview of the design and TAC calculations. A set of
penalties were defined to discourage non-sensible column designs, see appendix
B. The state of the environment was defined as the stream conditions, such as
temperature, pressure, molar flows, and revenue. The agent’s actionspace was
limited to setting the number of stages, feed stage, condenser pressure,
reflux -and reboil ratio. Boundaries of these parameters were set and are
shown in appendix C. To keep track of all streams, a stream table was created
which stores all data of a stream if the molar flow is greater or equal to 3.6
kmol/h and the purity specification is not met. Molar flows smaller then 3.6
kmol/h was regarded an outlet and was neither added to the stream table nor
was it sold. To decide which stream to separate next was based on the largest
molar flow present in the stream table. In order to compare the design
generated by the RL agent, two base cases were created by use of Aspen Plus
with optimisation via design specs and a random agent was programmed to take
random actions. The settings used to run the SAC agent are shown in appendix

##  5 Case results

###  5.1 Base case

For the base cases two separation sequences were considered, direct separation
or a tree-like sequence. Both these setups were able of generating profit
without generating any waste streams.

For the linear case, four columns were put in series to separate the feed
stream in its respective component. Appendix F shows a schematic of the
separation train including the column specifications. Each column was able to
produce a top and/or bottom stream with an equal or greater molar purity than
95 mol%. This result was gained with a stages range of 21 to 77 and a diameter
ranging from 4.9 to 10.4 meters. Optimised values for the reflux ratio and
distillate to feed ratio ranged from 2.72 to 11.28, and 0.35 to 0.67,
respectively. The TAC of the linear separation train was 107 M€ and a yearly
revenue of 1,552 M€. This results in a yearly profit of 1,445 M€.

In the case of a tree-like separation sequence two columns were put in series
and the last two columns were put in parallel. This configuration allows for
parallel processing of the top and bottom stream of the second column.
Appendix G shows the final column configuration with the columns respective
dimensions, TAC and revenue. The range of diameter and number of stages in
this configuration is from 6.5 to 9.5 meter and 21 to 85, respectively.
Optimisation of the columns was done with the build-in design spec option of
Aspen. For optimisation the distillate to feed ratio and reflux ratio were
chosen, since this directly impacts the desired product, the distillate. The
optimised values reflux ratio and distillate to feed ratio ranged from 1.09 to
8.56 and 0.34 to 0.77, respectively. A tree-like separation setup had a TAC of
107 M€, with a yearly revenue of 1,505 M€. This yields a profit of 1,398 M€.

###  5.2 SAC agent

For the SAC agent 2 cases will be evaluated, a maximum profit and a best
average case.

####  5.2.1 Max profit

Utilising the SAC agent with the settings provided in appendix E generated a
maximum profit of 1,311 M€ and an average profit of 384 M€. The maximum profit
was generated by utilising 8 columns, which are shown in appendix H.
Dimensions of the columns vary between 21 and 92 number of stages and a
diameter ranging from 5 to 9.9 meters. All open streams in this configuration
are at spec besides the bottom stream of column 4. This stream is not
considered an outlet, since the molar flow is greater then 3.6 kmol/h. Hence
the agent would continue separating this stream if it was allowed to. Figure 3
shows the trend in return of the RL agent. Here can be seen that there is
almost no increase in return after 3000 episodes.

Figure 3: Return per episode and moving average of RL agent yielding the
highest profit

Observing the agents’ actions for the first column provides a sense of how the
agent develops over time, because for the first column the input stream is
always the same. Figure 4 shows a converging trend for the reflux ratio over
episodes. This is indicative for the agent adjusting the parameters to
converge to an optimum. Appendix H shows the critic loss and entropy of the
agent. These plots together with the figure 4 indicate a fast learning agent
which takes little time to explore and start exploiting actions early on in
the process. Hence it would be desired to decrease the rate at which the agent
learns in order to provide sufficient time to explore new configurations.

Figure 4: Reflux ratio of first column of each episode

####  5.2.2 Best average

Increasing performance based on the average profit was found with increasing
the number of agent updates per episode to 4 and keeping the other parameters
similar to appendix E. This adjustment yielded an average profit of 457 M€ and
a maximum profit of 856 M€ within 9405 episodes. Figure 5 shows the trend in
return and moving average. At first the profit increases sharp, but decreases
as it continues. In comparison to the max profit case shows the adjustment a
steeper increase overall. This increase is observed, since the agent observes
the replay buffer more often. From figure 5 can be seen that the agent
converges towards until the 6500th episode. After this episode it starts
exploring again, hence the increased variance over returns. This trend was
also observed in the set column parameters.

Figure 5: Return per episode and a moving average of the RL agent yielding the
best average

Figure 6 shows the number of stages in the first column of each episode. It
shows the same converging trend until the 6500th episode after which it start
setting new values and gaining new experiences.

Figure 6: Number of stage of the first column of each episode

###  5.3 Overall

The stability and speed were tracked and are presented in table 1. From this
table can be seen that the RL agent almost always chooses to separate the
stream, which means that it takes every opportunity to gain experience.
Another good result is the low percentage of convergence error of the
flowsheet in Aspen. This indicates that the agent almost always takes actions
which are solvable by the flowsheet. However, the implementation of RL in
commercial simulation software showed some pitfalls with regard to speed and
stability. It took the agent 193 hours to complete 10,151 episodes. This
extensive time in combination with the unstable connection which rises RPC
errors makes for a tedious way of performing a sensitivity analysis of the
agents’ hyperparameters.

Table 1: Convergence statistics  Agent | Episodes | Duration | Separate
---|---|---|---
Yes | No
RL agent, standard | 10,151 | 193:43:13 | 99.8% | 0.2%
RL agent, 4 agent updates | 9405 | 160:58:43 | 98.9% | 1.1%
|  |  |  |
| Aspen runs
Agent | Lost contact | Convergence error | Converged
With warnings | Without warnings
RL agent, standard | 0.2% | 0.01% | 23.8% | 76.0%
RL agent, 4 agent updates |  | 0.2%
---
| 0.03%
---
| 35.4%
---
| 64.4%
---

##  6 Conclusion and future

This paper showed the applicability of RL in a chemical engineering context.
However, the RL agent was not able to outperform the created base case. To
potentially outperform the base case it is necessary to increase insight on
the effect of the hyperparameters. Based on the result in this study, the
agent stops exploring too early, therefore the learning of the agent should be
slowed down. Parameters which influence this learning are the learning rate of
$\alpha$ and the target smoothing parameter of the Q-target network. Lowering
these will lead to an increased learning time, but will allow for more
exploration by the agent.

To speed up the convergence to optimal solutions, good estimates can be passed
on to the agent by first simulating the column with a DSTWU in Aspen Plus and
feed the DSTWU column design to the SAC agent as initialisation.

The connection between Python and Aspen Plus was experienced as unstable and
Aspen Plus acts as a rather slow RL environment. Therefore it would be advised
to create a distillation model in Python, such that no external communication
is required.

## References

  * [1] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.  A general reinforcement learning algorithm that masters chess, shogi and Go through self-play.
  * [2] J. C. Hoskins and D. M. Himmelblau.  Process control via artificial neural networks and reinforcement learning.  Computers & Chemical Engineering, 16(4):241–251, 4 1992.
  * [3] Joohyun Shin, Thomas A. Badgwell, Kuang Hung Liu, and Jay H. Lee.  Reinforcement Learning – Overview of recent progress and implications for process control.  Computers & Chemical Engineering, 127:282–294, 8 2019.
  * [4] Laurence Midgley and Michael Thomson.  Reinforcement learning for chemical engineering process synthesis.  Bachelor Thesis, University of Cape Town, 11 2019.
  * [5] Laurence Illing Midgley.  Deep Reinforcement Learning for Process Synthesis.  https://arxiv.org/abs/2009.13265v1, 9 2020.
  * [6] Laura Stops, Roel Leenhouts, Qinghe Gao, and Artur M. Schweidtmann.  Flowsheet synthesis through hierarchical reinforcement learning and graph neural networks.  https://arxiv.org/abs/2207.12051, 7 2022.
  * [7] Quirin Göttl, Yannic Tönges, Dominik G. Grimm, and Jakob Burger.  Automated Flowsheet Synthesis Using Hierarchical Reinforcement Learning: Proof of Concept.  Chemie Ingenieur Technik, 93(12):2010–2018, 12 2021.
  * [8] Quirin Göttl, Dominik G. Grimm, and Jakob Burger.  Automated synthesis of steady-state continuous processes using reinforcement learning.  Frontiers of Chemical Science and Engineering, 16(2):288–302, 2 2022.
  * [9] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  MIT Press, Cambridge, second edition, 2018.
  * [10] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft Actor-Critic Algorithms and Applications.  https://arxiv.org/abs/1812.05905, 12 2018.

## Appendix

##  Appendix A Feed stream

Table 2: Feed stream specifications Component | Mol%
---|---
$C_{2}H_{6}$ | 0.06
$C_{3}H_{8}$ | 33.69
i$C_{4}H_{10}$ | 35.65
n$C_{4}H_{10}$ | 15.32
i$C_{5}H_{12}$ | 10.21
n$C_{5}H_{12}$ | 5.13
Molar flow | [kmol/h] | 12,400
Temperature | [${}^{\circ}C$] | 105
Pressure | [bar] | 17

##  Appendix B Agent penalties

Table 3: Agent penalties Parameter | Non-sensible condition | Penalty |  |  |  |
---|---|---|---|---|---|---
Qcondenser |  $\geq$ 0 | 5 M€ |  |  |  |
Qreboiler |  $\leq$ 0 | 5 M€ |  |  |  |
Treboiler |  $\geq$ Tsteam | 5 M€ |  |  |  |
$\Delta T_{lm_{condenser}}$ | $\leq T_{cool}$ | 5 M€ |  |  |  |
Flowsheet error | - | 10 M€ |  |  |  |

##  Appendix C Action boundaries

Table 4: Parameter boundaries Parameter | Type | Bound
---|---|---
Min | Max
Number of stages | Discrete | 20 | 100
Feed stage | Discrete | 0.2*nstages | 0.8*nstages
Condenser pressure | Real | 0.5 | 30
Reflux ratio | Real | 1 | 20
Boilup ratio | Real

##  Appendix D Equations

Table 5: Design and economical equations Design equations |  |  |
---|---|---|---
Column diameter
$D=1.1*\max(D_{eff})$
Where |  |  |
$D$ | = | diameter | [$m$]
$D_{eff}$ | = | effective diameter | [$m$]
$n_{s}$ | = | 0.2 < $n_{s}$ <0.8 | [$-$]
$D_{eff}=\sqrt{\frac{4*\Phi_{mol,n_{s}}^{vapor}}{3.1416*f}*\sqrt{\frac{R*T_{s}*Mw_{n_{s}}^{vapor}*1000}{P}}}$
$\Phi_{mol,n_{s}}^{vapor}$ | = | molar vapor flow rate | [$\frac{mol}{s}$]
$f$ | = | gas load factor (fixed, 1.6) | [$Pa^{0.5}$]
$R$ | = | molar gas factor (fixed, 1.6) | [$\frac{J}{mol*K}$]
$T_{n_{s}}$ | = | temperature | [$K$]
$Mw_{n_{s}}^{vapor}$ | = | vapor molar weight | [$\frac{g}{mol}$]
P | = | column pressure | [$Pa$]
Column height
$L=n_{s}*HETP+H_{0}$
Where |  |  |
L | = | column height | [$m$]
HETP | = | height equivalent to a theoretical plate | [$m$]
$H_{0}$ | = | clearance (fixed, 4) | [$m$]
Condenser area
$A_{cnd}=-\frac{Q_{cnd}}{K_{cnd}*\Delta T_{lm,cnd}}$
Where |  |  |
$A_{cnd}$ | = | condenser area | [$m^{2}$]
$Q_{cnd}$ | = | condenser duty, negative value | [$W$]
$K_{cnd}$ | = | heat transfer coefficient (fixed, 500) | [$\frac{W}{m^{2}*^{\circ}C}$]
$\Delta T_{lm,cnd}$ | = | logarithmic mean temperature difference | [°$C$]
$\Delta
T_{lm,cnd}=\sqrt[3]{\frac{(T_{cnd}-T_{in}^{cool})*(T_{cnd}-T_{out}^{cool})*((T_{cnd}-T_{in}^{cool})+(T_{cnd}-T_{out}^{cool}))}{2}}$
Where |  |  |
$T_{cnd}$ | = | condenser temperature | [°$C$]
$T_{in}^{cool}$ | = | supply temperature cooling water (fixed, 30) | [°$C$]
$T_{out}^{cool}$ | = | return temperature cooling water (fixed, 40) | [°$C$]
Reboiler area
---
$A_{rbl}=\frac{Q_{rbl}}{K_{rbl}*\Delta T_{lm,rbl}}$
Where |  |  |
$A_{rbl}$ | = | reboiler area | [$m^{2}$]
$Q_{rbl}$ | = | reboiler duty, positive value | [$W$]
$K_{rbl}$ | = | heat transfer coefficient (fixed, 800) | [$\frac{W}{m^{2}*^{\circ}C}$]
$\Delta T_{lm,rbl}$ | = | temperature difference | [°$C$]
$\Delta T_{lm,rbl}=T_{steam}-T_{rbl}$
Where |  |  |
$T_{steam}$ | = | steam temperature (fixed, 201) | [°$C$]
$T_{rbl}$ | = | reboiler temperature | [°$C$]
Investment cost
$I=F_{cap}*C_{inv}$
Where |  |  |
$I$ | = | investment | [$M$€]
$F_{cap}$ | = | capital charge factor | [$-$]
$C_{inv}$ | = | investment cost | [$M$€]
$C_{inv}=F_{L}*C_{eqp}$
$F_{L}$ | = | Lang factor (fixed, 5) | [$-$]
Equipment cost
$C_{eqp}=\frac{C_{col}+C_{int}+C_{cnd}+C_{rbl}}{1000000}$
$C_{eqp}$ | = | equipment cost | [$M$€]
$C_{col}$ | = | column cost | [€]
$C_{int}$ | = | internals cost | [€]
$C_{cnd}$ | = | condenser cost | [€]
$C_{rbl}$ | = | reboiler cost | [€]
Column cost
$C_{col}=\frac{0.9*Ms_{2018}*937.64*D^{1.066}*L^{0.802}*F_{c,col}}{280}$
$C_{col}$ | = | column cost | [€]
$Ms_{2018}$ | = | Marshall & Swift equipment index 2018 (fixed, 1638.2) | [$-$]
$F_{c,col}$ | = | correction factor for column | [$-$]

$F_{c,col}=F_{m}+F_{p}$
---
$F_{m}$ | = | correction factor for column shell material (fixed, 1) | [$-$]
$F_{p}$ | = | correction factor for column pressure (fixed, 1) | [$-$]
Internal costs
$C_{int}=\frac{0.9*Ms_{2018}*97.24*D^{1.55}*L*F_{c,int}}{280}$
$C_{int}$ | = | internal cost | [€]
$F_{c,int}$ | = | correction factor for internals | [$-$]
$F_{int,c}=F_{int,s}+F_{int,t}+F_{int,m}$
$F_{int,s}$ | = | correction factor for tray spacing (fixed, 1.4) | [$-$]
$F_{int,t}$ | = | correction factor for tray type (fixed, 0) | [$-$]
$F_{int,m}$ | = | correction factor for internals material (fixed, 0) | [$-$]
Condenser cost
$C_{cnd}=\frac{0.9*Ms_{2018}*474.67*A_{cnd}^{0.65}*F_{c,cnd}}{280}$
$C_{cnd}$ | = | condenser cost | [€]
$F_{cnd,c}$ | = | correction factor for condenser | [$-$]
$F_{cnd,c}=(F_{htx,P}+F_{htx.d})*F_{htx,m}$
$F_{htx,P}$ | = | correction factor for pressure (fixed, 0) | [$-$]
$F_{htx.d}$ | = | correction factor for design type: fixed - tube sheet (fixed, 0.8) | [$-$]
$F_{htx,m}$ | = | correction factor for material (fixed, 1) | [$-$]
Reboiler cost
$C_{rbl}=\frac{0.9*Ms_{2018}*474.67*A_{rbl}^{0.65}*F_{c,rbl}}{280}$
$F_{rbl,c}$ | = | condenser duty, negative value | [$W$]
$F_{rbl,c}=(F_{htx,P}+F_{htx,d})*F_{htx,m}$
Condenser operation cost
$C_{cnd,operation}=\frac{E_{cnd}*E_{cost,cnd}}{1,000,000}$
$C_{cnd,operation}$ | = | operational cost condenser | [$M$€]
$E_{cnd}$ | = | condenser energy | [$\frac{kWh}{year}$]
$E_{cost,cnd}$ | = | energy price | [$\frac{€}{kWh}$]
$E_{cnd}=\frac{uptime*Q_{cnd}}{1000}$
$E_{cost,cnd}=6*10^{-6}*T_{cool}^{2}-0.0006*T_{cool}+0.0163$

Reboiler operation cost
---
$C_{rbl,operation}=\frac{Q_{rbl}*M*c_{steam}*3600*uptime}{\Delta
H_{vap}*1,000,000}$
$C_{rbl,operation}$ | = | operational costs reboiler | [$M$€]
$M$ | = | molar weight of water (fixed, 18) | [$\frac{g}{mol}$]
$c_{steam}$ | = | steam price at 16 bar (fixed, 18) | [$\frac{\text{€}}{ton}$]
$\Delta H_{vap}$ | = | molar heat of vaporization of 16 bar steam (fixed, 34794) | [$\frac{J}{mol}$]

##  Appendix E Reinforcement learning agent parameters

Table 6: Reinforcement learning agent parameters Agent | Max profit | Best average
---|---|---
Optimiser | Adam
Learning rate | 3e-4
Discount factor | 0.99
Temperature | Adaptive
Target smoothing coefficient | 0.005
Target entropy | -5 (=-dim[A])
Replay buffer size |  | Min. 320,
---
Max. 320.000
Reward scale | 10
Network updates per episode | 1 | 4
Network |  |
Hidden layers | 2
Hidden layer size | 128
Type of network | FeedForward

##  Appendix F Base case results - Linear

Figure 7: Base case linear separation sequence Table 7: Linear column specifications with optimised reflux ratio and distillate/feed ratio Column | Unit | 1 | 2 | 3 | 4
---|---|---|---|---|---
Condenser pressure | [bar] | 17.2 | 7.2 | 7.2 | 2.0
Reflux ratio | [-] | 11.28 | 5.24 | 2.72 | 5.04
Boilup ratio | [-] | 4.07 | 5.76 | 3.31 | 10.64
Distillate/Feed | [-] | 0.35 | 0.55 | 0.49 | 0.67
Condenser duty | [MW] | -181 | -130 | -34 | -49
Reboiler duty | [MW] | 130 | 114 | 34 | 44
Table 8: Stream table linear base case | Molar purity
---
mol%
Stream number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
C2H6 | 0.1 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
C3H8 | 33.2 | 95.0 | 0.5 | 0.9 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
iC4H10 | 35.8 | 4.6 | 52.4 | 95.0 | 1.1 | 2.3 | 0.0 | 0.0 | 0.0
nC4H10 | 15.4 | 0.2 | 23.5 | 4.1 | 46.8 | 95.0 | 0.9 | 1.4 | 0.0
iC5H12 | 10.3 | 0.0 | 15.8 | 0.0 | 34.7 | 2.4 | 65.4 | 95.0 | 5.0
nC5H12 | 5.2 | 0.0 | 7.9 | 0.0 | 17.4 | 0.3 | 33.7 | 3.6 | 95.0
| Molar flow
---
kmol/h
Stream number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
C2H6 | 6 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0
C3H8 | 3995 | 3955 | 40 | 40 | 0 | 0 | 0 | 0 | 0
iC4H10 | 4312 | 192 | 4120 | 4079 | 40 | 40 | 0 | 0 | 0
nC4H10 | 1857 | 10 | 1847 | 174 | 1673 | 1656 | 17 | 17 | 0
iC5H12 | 1240 | 0 | 1240 | 0 | 1240 | 42 | 1198 | 1168 | 30
nC5H12 | 623 | 0 | 623 | 0 | 623 | 5 | 618 | 45 | 574
Total | 12033 | 4164 | 7869 | 4293 | 3576 | 1743 | 1833 | 1229 | 604
Temperature [°C] | 105 | 51.6 | 106 | 66.1 | 52.2 | 80.5 | 65.3 | 102 | 51.8 | 48.6 | 57.8
Pressure [bar] | 17.2 | 17.2 | 17.2 | 7.2 | 7.2 | 7.2 | 7.2 | 7.2 | 2.0 | 2.0 | 2.0

##  Appendix G Base case results - Tree

Figure 8: Base case tree-like separation sequence Table 9: Tree-like column specifications with optimised reflux ratio and distillate/feed ratio Column | Unit | 1 | 2 | 3 | 4
---|---|---|---|---|---
Condenser pressure | [bar] | 17.2 | 7.2 | 6.7 | 2.0
Reflux ratio | [-] | 8.6 | 1.1 | 4.7 | 6.7
Boilup ratio | [-] | 2.7 | 4.8 | 13.3 | 13.4
Distillate/Feed | [-] | 0.34 | 0.77 | 0.71 | 0.66
Condenser duty | [MW] | -139 | -64 | -126 | -61
Reboiler duty | [MW] | 87 | 49 | 125 | 56
Table 10: Stream table tree-like base case | Molar purity
---
[mol%]
Stream number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
C2H6 | 0.1 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
C3H8 | 33.2 | 95.0 | 1.2 | 1.6 | 0.0 | 2.3 | 0.0 | 0.0 | 0.0
iC4H10 | 35.8 | 4.6 | 52.0 | 67.3 | 0.1 | 95.0 | 0.7 | 0.2 | 0.0
nC4H10 | 15.4 | 0.3 | 23.3 | 29.8 | 1.0 | 2.7 | 95.0 | 1.5 | 0.0
iC5H12 | 10.3 | 0.0 | 15.6 | 1.2 | 64.5 | 0.0 | 4.1 | 95.0 | 5.0
nC5H12 | 5.2 | 0.0 | 7.9 | 0.1 | 34.3 | 0.0 | 0.2 | 3.3 | 95.0
| Molar flow
---
[kmol/h]
Stream number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
C2H6 | 6 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0
C3H8 | 3995 | 3897 | 99 | 99 | 0 | 99 | 0 | 0 | 0
iC4H10 | 4312 | 189 | 4123 | 4121 | 2 | 4107 | 13 | 2 | 0
nC4H10 | 1857 | 12 | 1845 | 1826 | 18 | 118 | 1709 | 18 | 0
iC5H12 | 1240 | 0 | 1239 | 73 | 1166 | 0 | 73 | 1136 | 31
nC5H12 | 623 | 0 | 623 | 3 | 620 | 0 | 3 | 39 | 581
Total | 12033 | 4104 | 7929 | 6122 | 1807 | 4324 | 1799 | 1195 | 611
Temperature [°C] | 105 | 51.6 | 105 | 65.0 | 55.3 | 51.3 | 102 | 51.8 | 48.5 | 62.9 | 48.4 | 57.8
Pressure [bar] | 17.2 | 17.2 | 17.2 | 7.2 | 7.2 | 6.7 | 7.2 | 2.0 | 6.7 | 6.7 | 2.0 | 2.0

##  Appendix H Reinforcement learning agent results - max profit

Figure 9: Reinforcement learning separation sequence - max profit Figure 10:
Critic loss - max profit Figure 11: Entropy - max profit

##  Appendix I Reinforcement learning agent results - best average

Figure 12: Critic loss - Best average Figure 13: Entropy - best average


# Light-shift-free and dead-zone-free atomic orientation based scalar
magnetometry using a single amplitude-modulated beam

Q.-Q. Yu  S.-Q. Liu  C.-Q. Yuan  D. Sheng<EMAIL_ADDRESS>Department of
Precision Machinery and Precision Instrumentation, Key Laboratory of Precision
Scientific Instrumentation of Anhui Higher Education Institutes, University of
Science and Technology of China, Hefei 230027, China

###### Abstract

Detection dead zones and heading errors induced by light shifts are two
important problems in optically pumped scalar magnetometry. We introduce an
atomic orientation based single-beam magnetometry scheme to simultaneously
solve these problems, using a polarization-reversing and path-bending Herriott
cavity. Here, a reflection mirror is inserted into the cavity to bend the
optical paths in the middle, and divide them into two separated orthogonal
regions to avoid the detection dead zone. Moreover, half-wave plates are added
in the center of each optical region, so that the light polarization is
flipped each time it passes the wave plates and the light shift effects are
spatially averaged out. This operation is demonstrated to eliminate the
unnoticed heading errors induced by ac light shifts. The methods developed in
this paper are robust to use, and easy to be applied in other atomic devices.

##  I Introduction

Optically pumped scalar magnetometers resolve atomic Zeeman splittings to
extract the bias field magnitude Smullin  _et al._ (2009). With developments
in experiment schemes, hardware and techniques, scalar magnetometers have
reached great advances in sensitivities Sheng  _et al._ (2013); Lucivero  _et
al._ (2021a, b), stabilities Wilson  _et al._ (2019), and miniaturization
Kitching (2018). They have been widely used in researches on fundamental
physics Abel  _et al._ (2020); Afach  _et al._ (2021), geophysics Prouty  _et
al._ (2013), space science Acuna (2002), and neuroscience Limes  _et al._
(2020); Zhang  _et al._ (2020). However, due to working principles, they
usually suffer two important problems in practical applications, which are
detection dead zones and heading errors.

As initially studied by Bloom Bloom (1962), detection dead zones in scalar
magnetometers refer to certain directions of the target field where the sensor
signal-to-noise ratio is significantly reduced. For example, the detection
dead zones of magnetometers using dc pump beams involve regions perpendicular
to the pump beam direction, and the dead zones of double-resonance
magnetometers include extra regions near the radio-frequency excitation field
direction. Several methods have been developed to solve this problem,
including the use of pulse laser to increase the pump beam power Limes  _et
al._ (2020), employing multiple vapor cells Acuna (2002); B. Chéron and Hamel
(2001), and combining multi-harmonic information Wang  _et al._ (2021).

Heading errors correspond to the effect that the measured field magnitude is
correlated with the sensor orientation. Nonlinear Zeeman effects Bao  _et al._
(2018); Lee  _et al._ (2021) and light shifts Oelsner  _et al._ (2019) are two
main sources of such errors, where the latter one can be the dominant factor
when the bias field is not large. In optical pumping experiments, light shifts
correspond to the changes of atomic ground state transition frequencies due to
interactions between atoms with light beams Cohen-Tannoudji (1962); Happer
(1972). There are two physical mechanism to generate such shifts: one is due
to the transfer of coherence between ground and excited states in atomic
transitions Cohen-Tannoudji (1962); Norris  _et al._ (2012), and the other one
is due to ac Stark shifts Mathur  _et al._ (1968); Happer (1972). In atomic
cell experiments filled with quenching gases, the former one is largely
suppressed Appelt  _et al._ (1998). The ac Stark shift can be divided into
scalar, vector and tensor parts Happer (1972), among which the vector part is
the most important one in atomic orientation based magnetometers. This vector
light shift is determined by the circular polarization part of the light, and
its effect on atoms is equivalent to an effective magnetic field Mathur  _et
al._ (1968). It is common to circumvent this problem by either locking the
beam on resonance, or employing the atomic alignment based magnetometry Abel
_et al._ (2020); Zhang  _et al._ (2022).

In miniaturized atomic devices Kitching (2018), it is often preferred to leave
the laser wavelength unlocked, so that both the power and space can be saved.
Moreover, in certain atomic orientation based magnetometry, such as the one
using a single elliptically polarized light Shah and Romalis (2009), the
optical pumping beam has to be kept off resonance. In these cases, it requires
to develop new methods and schemes to simultaneously eliminate the detection
dead zones and light shift, even when the light frequency is off resonance or
drifting. A previous successful scheme employs a single polarization
modulating beam, which can simultaneously excite orientation and alignment
resonances Ben-Kish and Romalis (2010). While the combination of these two
kinds of resonances can eliminate the detection dead zones, the light shift
effect is time averaged out. However, in practice, the electrical-optical
modulator used for polarization modulations is sensitive to the environment
parameters. This not only is a problem for the device calibration, but also
can cause drifts or fluctuations of the time-averaged light shift result.

In this work, we present an atomic orientation based scalar magnetometer,
which only uses a single amplitude-modulated beam. This sensor is assisted by
a Herriott cavity, which is modified to lift the detection dead zones and
suppress light shifts. Following this introduction, Sec. II introduces the
setup and working principle of the magnetometer employing a conventional
Herriott cavity, Sec. III presents the measurement results and identifies the
problems in this conventional magnetometer, Sec. IV. describes the solutions
of these problems by modifying the optical properties of the beams inside the
cavity, and Sec. V concludes the paper.

##  II Magnetometer setup and working principle

A Herriott-cavity-assisted vapor cell, which is made by the anodic bonding
technique Cai  _et al._ (2020); Hao  _et al._ (2021), is used for the atomic
magnetometer. This cavity consists of two cylindrical mirrors with a curvature
of 100 mm, a diameter of 12.7 mm, a thickness of 2.5 mm, and a separation of
19.3 mm. In this work, the multipass cavity increases the interaction length
between light and atoms Li  _et al._ (2011) so that the cell can be kept at a
relatively low working temperature, which is essential for the stability of
some optical elements as discussed in the latter part of the paper. Such a
cavity also provides a platform for manipulating the optical properties of
beams inside. The vapor cell, filled by Rb atoms with natural abundances and
350 torr of N2 gas, is placed in a 3D printed optical platform, and heated to
a temperature around 55∘C by running ac currents through ceramic heaters.

Figure 1 shows the optical platform used for a single-beam Bell-Bloom scalar
magnetometer Bell and Bloom (1961). The pump beam is amplitude modulated using
acoustic-optical modulators with a duty cycle of 20$\%$, and fiber coupled to
the optical platform. This pump beam, with a diameter of 1 mm and a circular
polarization, enters the cavity from a 2.5 mm hole in the center of the front
mirror, and exits from the same hole after 21 times of reflections. This
magnetometer sensor sits in the middle of five-layer mu-metal shields, with a
10 $\mu$T bias field generated by solenoid coils inside the shields. The
sensor is also connected to a rotation table outside the shields through a
plastic holder, so that the relative angle between the pump beam direction and
the bias field can be precisely controlled.

Two photodiode signals are sent out from the sensor, one is the transmitted
pump beam signal, and the other one is the reference signal by recording a
part of the input beam before it enters the cavity. This latter one is also
used to lock the time averaged input beam power to 0.14 mW. The current
signals from these detectors are first converted to voltage signals by two
transimpedance amplifiers with feedback resistors of 10 k$\Omega$ and
bandwidths around 100 Hz. These voltage signals are sent to a differential
amplifier, with a unit gain on the pump beam signal channel and a variable
gain $g$ on the reference signal channel. The subtractor output is further
amplified by a factor of 200, and recorded as the magnetometer signal.

Figure 1: (Color online) Illustration of the optical platform for a single-
beam Bell-Bloom scalar magnetometer. PBS: polarization beam splitter, PD:
photodiode detector.

Two coordinates are used to describe the spin dynamics following the notations
in Ref. Cai  _et al._ (2020). One is the $xyz$ coordinate with the $z$ axis
along the center of pump beam directions. The other one is
$x^{\prime}y^{\prime}z^{\prime}$ coordinate with the $z^{\prime}$ axis along
the bias field direction, and $y^{\prime}$ axis perpendicular to the
$zoz^{\prime}$ plane. We denote the angle between the $z$ and $z^{\prime}$
axes as $\psi_{z}$. For this single-beam scalar magnetometer, we fix the $y$
axis in the $zoz^{\prime}$ plane as shown in the inset of Fig. 1. The dynamics
of the electron spin polarization $\boldsymbol{P}$ is described by the Bloch
equation Shah and Romalis (2009):

$~{}\frac{d\boldsymbol{P}}{dt}=\gamma\boldsymbol{P}\times(\boldsymbol{B}_{0}+\boldsymbol{B}_{L})+\frac{1}{Q(P)}[R_{OP}(\boldsymbol{s}-\boldsymbol{P})-R_{d}\boldsymbol{P}],$
(1)

where $\gamma$ is the atomic gyromagnetic ratio, $Q(P)$ is the nuclear spin
slowing down factor, $R_{OP}$ is the optical pumping rate, $\boldsymbol{s}$ is
the photon spin of the light beam Appelt  _et al._ (1998), $R_{d}$ is the atom
depolarization rate in the absence of light, and $\boldsymbol{B}_{L}$ is the
effective field induced by the light shift effect. $B_{L}$ is connected to
$R_{OP}$ by the relation Appelt  _et al._ (1999); Savukov and Romalis (2005)

$~{}R_{OP}s+i(2I+1)\gamma B_{L}=\frac{r_{e}cf\Phi s/Ar}{\Gamma-i\Delta\nu},$
(2)

with $I$ as the nuclear spin, $r_{e}$ as the classical electron radius, $c$ as
the speed of light, $f$ as the oscillator strength, $\Phi$ as the light photon
flux, $Ar$ as the beam area, $\Gamma$ as the atomic transition line width, and
$\Delta\nu$ as the light detuning.

For a pump beam which is amplitude modulated at a frequency $\omega$, the
optical pumping rate can be expressed as

$~{}R_{OP}=a_{0}+\sum_{n=1}^{\infty}a_{n}\cos(n\omega t-\alpha_{n}),$ (3)

where $a_{i}$ is the corresponding coefficient of the Fourier expansion series
of $R_{OP}$. With the pump beam duty cycle of 20% in this experiment, we have
$a_{1}/a_{0}\simeq 1.9$. When the pump beam is on resonance with the Rb D1
transition and $\omega$ is close to the atomic Larmor precession frequency
$\omega_{L}$, a substantial transverse atomic polarization can be built as
Bell and Bloom (1961); Cai  _et al._ (2020)

$~{}P_{x^{\prime}}+iP_{y^{\prime}}=\frac{sr_{n}a_{1}\sin\psi_{z}}{2[R+iQ(P)(\omega_{L}-\omega)]}e^{-i(\omega
t-\alpha_{1})},$ (4)

where $R=R_{d}+a_{0}$, $r_{n}$ is the abundance of the Rb isotope that is
driven by the modulated pump beam. In most situations, $\omega\gg R$ and the
ac part of longitudinal atomic polarization is negligible compared with
$P_{x^{\prime}}$ Cai  _et al._ (2020). The static part of the longitudinal
polarization is:

$~{}P_{z^{\prime}}=sa_{0}\cos\psi_{z}/(R_{d}+a_{0})$ (5)

The propagation equation of the pump beam intensity is:

$~{}\frac{dI(z)}{dz}=-I(z)[1-\boldsymbol{s}\cdot{\boldsymbol{P}(z)}]n\sigma,$
(6)

where $\sigma$ is the photon absorption cross section of atoms, and $n$ is the
atom density. Taking the time average, we have:

$\displaystyle~{}\frac{d\langle I(z)\rangle}{dz}=-\langle I(z)\rangle
n\sigma\left[1-sP_{z^{\prime}}(z)\cos\psi_{z}-\right.$
$\displaystyle\left.\frac{sr_{n}a_{1}(z)\sin\psi_{z}}{2a_{0}(z)}A({P}_{x^{\prime},s}(\omega))\right],$
(7)

where the operator $\langle\cdots\rangle$ denotes the time average, $A(P)$ is
the amplitude of the oscillating parameter $P$, and $P_{x^{\prime},s}$ is the
part of $P_{x^{\prime}}$ that is in synchronization with the first harmonic of
the pump beam modulations in Eq. (3),

$~{}P_{x^{\prime},s}(\omega)=\frac{sa_{1}\cos(\omega
t-\alpha_{1})\sin\psi_{z}}{2}\frac{R}{R^{2}+Q^{2}(P)(\omega_{L}-\omega)^{2}}.$
(8)

The analytic solution of Eq. (II) is the principal value of the Lambert W
function Kornack (2005). A more convenient way to effectively express the
transmitted beam intensity $I_{t}$ is:

$~{}\langle I_{t}\rangle=c\langle I_{0}\rangle
e^{-N_{d}}e^{N_{d}\left[s\overline{P}_{z^{\prime}}\cos\psi_{z}+sr_{n}\sin\psi_{z}A(\overline{P}_{x^{\prime},s}(\omega))\overline{a}_{1}/2\overline{a}_{0}\right]},$
(9)

where $I_{0}$ is the input beam intensity, the coefficient $c$ denotes the
ratio of $I_{t}/I_{0}$ without atoms, and $c$ is determined by the optical
patterns and beam power loss inside the cavity. The optical depth $N_{d}$ is
equal to $n\sigma l$, with $l$ as the length of optical pathes inside the
cell, and $\overline{x}$ is the spatial average of the parameter $x$ over the
whole optical pathes.

##  III Magnetometer results and problems

In this scalar magnetometer, the bias field amplitude is extracted from the
atomic Larmor frequency, which is determined by finding the resonant response
of atoms to $\omega$. Figure 2 shows the experiment results when the pump beam
is resonant with the Rb D1 line. Here, the recorded signal can be expressed as
$V=G(\langle I_{t}\rangle-g\langle I_{ref}\rangle)$, where $I_{ref}$ is the
reference beam intensity recorded by one of the photodiode detectors in the
sensor, $g$ is the variable gain of the reference signal channel in the
differential amplifier mentioned previously, and $G$ is the converting factor
that connects the recorded beam intensity and the voltage output from the
sensor electronics. The data shows symmetrical line shapes, which agrees with
predictions from Eqs. (8) and (9). The exact line shape of the Herriott-
cavity-assisted magnetometer signal is complex, especially due to the
diffusion of atoms among the complicated optical patterns Xiao  _et al._
(2006); Sheng  _et al._ (2013); Lucivero  _et al._ (2017). It has been shown
that Li  _et al._ (2011) we can use a combination of two Lorentzian functions
to describe the atomic polarization in the frequency domain. Therefore, the
magnetometer signal can be fitted by the following equation:

$~{}f(\omega)=h\exp\left[\sum_{i=1}^{2}\frac{c_{i}(\frac{\Gamma_{i}}{2})^{2}}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\right]+b.$
(10)

Figure 2 shows good fitting results using Eq. (10).

Figure 2:  Illustration of the magnetometer signal as a function of the pump
beam amplitude modulation frequency while the pump beam is on resonance with
the Rb D1 line and $\psi_{z}=30^{\circ}$. Here we focus on 85Rb atoms, as
follows in the rest of the paper. The solid line is the fitting result using
Eq. (10).

To characterize the magnetometer signal amplitude, we define a parameter $S$
as the difference between the magnetometer resonant signal
($\omega=\omega_{L}$) and the signal when $\omega$ is far from $\omega_{L}$.
In the case that the light is resonant with the Rb D1 line, $S$ is determined
from Eq. (9):

$\displaystyle~{}S$ $\displaystyle\propto$ $\displaystyle\langle
I_{t}(\omega=\omega_{L})\rangle-\langle I_{t}(\omega\rightarrow\infty)\rangle$
(11) $\displaystyle\propto$
$\displaystyle\exp\left[-N_{d}+N_{d}\frac{s^{2}\overline{a_{0}}}{R_{d}+\overline{a_{0}}}\cos^{2}\psi_{z}\right]\times$
$\displaystyle\left[\exp\left(N_{d}\frac{s^{2}r_{n}\overline{a_{1}}}{4\overline{a_{0}}}\frac{\overline{a_{1}}}{R_{d}+\overline{a_{0}}}\sin^{2}\psi_{z}\right)-1\right].$

When the cell temperature is low enough, the optical depth $N_{d}$ is
negligible and the atomic polarization part in Eq. (II) can be neglected, so
that the transmitted beam intensity is almost independent of the atom
polarization. This is the situation of the experiment data at cell temperature
of 35∘C in Fig. 3, where the transmitted beam power has weak angular
dependence as shown in Fig. 3 (a). In this case, $S$ is determined by the last
line in Eq. (11), which predicts that $S$ is maximum at $\psi_{z}=90^{\circ}$
and decreases with $\psi_{z}$. This is consistent with the experiment results
at the same cell temperature in Fig. 3 (b). As the cell temperature and
$N_{d}$ increases, the atomic polarization part starts to play an important
role to determine the transmitted beam power. This case corresponds to the
experiment data at cell temperature of 55∘C in Fig. 3. Due to the
$\cos^{2}\psi_{z}$ term in Eq. (11), the maximum amplitude of $S$ shifts from
$\psi_{z}=90^{\circ}$ as shown in Fig. 3 (b). While the two experiment
conditions lead to quite different experiment results when $\psi_{z}$ is
large, the magnetometer signal amplitudes show similar rapid decay as
$\psi_{z}$ approaches to zero. This is due to the last line in Eq. (11), which
is proportional to $\sin^{2}\psi_{z}$ when $\psi_{z}$ is small. Therefore, the
regions near $\psi_{z}=0^{\circ}$ are detection dead zones of current
magnetometers.

Figure 3: (Color online) Plot (a) and (b) shows the transmitted beam power and
the $S$ value of the magnetometer response as a function of the $\psi_{z}$ at
two different cell temperature.

To better compare the experiment results in different experiment conditions,
we further define the normalized magnetometer response as the original
magnetometer signals modified by subtracting the signal when $\omega$ is far
off resonance, and then normalized by the parameter $S$. For the data in Fig.
2, its corresponding normalized magnetometer response can be described by:

$\displaystyle~{}N(\omega)$ $\displaystyle=$
$\displaystyle\frac{f(\omega)-(h+b)}{S}$ (12) $\displaystyle=$
$\displaystyle\frac{\exp\left[\sum_{i=1}^{2}\frac{c_{i}(\frac{\Gamma_{i}}{2})^{2}}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\right]-1}{\exp(\sum_{i=1}^{2}c_{i})-1},$

where the parameters in the equation are the same as Eq. (10). Figure 4 (a)
shows the normalized magnetometer responses with three different $\psi_{z}$ at
a cell temperature around 55 ∘C. While all the results have symmetric line
shapes, the line width increases as $\psi_{z}$ decreases. This is due to fact
that, as shown in Fig. 3 (a), the transmitted beam power increases when
$\psi_{z}$ changes from 90∘ to $30^{\circ}$, which leads to a larger power
broadening effect.

Figure 4: (Color online) (a) Normalized magnetometer responses at different
$\psi_{z}$, when the pump beam is resonant with the Rb D1 line and the cell
temperature is around 55 ∘C. The solid lines are fitting results using Eq.
(12). Plot (b) shows the same results as plot (a), except that the pump beam
is 6 GHz blue detuned from resonance. The solid lines are fitting results
using Eq. (15).

However, the situation becomes more complicated when the pump beam is off
resonance. In this case, we need to consider the light shift effects. Similar
to $R_{OP}$, the effective field from light shifts can be expanded as
$\boldsymbol{B}_{L}=[B_{L0}+\sum_{n}B_{Ln}\cos(n\omega
t-\alpha_{n})]\hat{\boldsymbol{z}}$. The first harmonic part of
$\boldsymbol{B}_{L}$ can excite a part of $P_{z^{\prime}}$ to the
$x^{\prime}y^{\prime}$ plane, so that there is an additional polarization term
$P^{\prime}_{x^{\prime}}$ along the $x^{\prime}$ axis besides $P_{x^{\prime}}$
in Eq. (4). By treating this effect as a perturbation, we can express
$P^{\prime}_{x^{\prime}}$ as Seltzer (2008):

$\displaystyle~{}P_{x^{\prime}}^{\prime}=\frac{(\omega_{L}-\omega)\gamma
B_{L1}\cos(\omega
t-\alpha_{1})\sin\psi_{z}P_{z^{\prime}}}{2[R^{2}/Q^{2}(P)+(\gamma
B_{L1}\sin\psi_{z}/2)^{2}+(\omega_{L}-\omega)^{2}]}.$ (13)

The transmitted pump beam intensity is now determined by
$A(\overline{P}_{x^{\prime},s}+\overline{P^{\prime}}_{x^{\prime}})$, instead
of $A(\overline{P}_{x^{\prime},s})$, in Eq. (9). Therefore, the dispersion
relation between $P_{x^{\prime}}^{\prime}$ and $\omega$ introduces asymmetric
magnetometer response results. Similar to Eq. 10, a combination of two
Lorentzian functions and two dispersion functions are used to describe the
atomic polarization behaviors. The function to fit the magnetometer signal
with detuned light is:

$\displaystyle~{}F(\omega)=h\exp\left[\sum_{i=1}^{2}\frac{c_{i}(\frac{\Gamma_{i}}{2})^{2}+d_{i}(\omega-\omega_{0})}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\right]+b,$
(14)

and the normalized response is described by:

$\displaystyle~{}N^{\prime}(\omega)$ $\displaystyle=$
$\displaystyle\frac{F(\omega)-(h+b)}{S}$ (15) $\displaystyle=$
$\displaystyle\frac{\exp\left[\sum_{i=1}^{2}\frac{c_{i}(\frac{\Gamma_{i}}{2})^{2}+d_{i}(\omega-\omega_{0})}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\right]-1}{\exp(\sum_{i=1}^{2}c_{i})-1}.$

Figure 4 (b) shows the normalized magnetometer results, when the pump beam is
6 GHz blue detuned from the D1 line. It confirms the appearance of
asymmetrical line shapes as $\psi_{z}$ deviates from 90∘. On the other hand,
the data also shows that the asymmetry of the line shapes increases as
$\psi_{z}$ decreases. This is because that the degree of this asymmetry is
determined by the ratio
$A(\overline{P^{\prime}}_{x^{\prime}})/A(\overline{P}_{x^{\prime},s})$, which
is proportional to $\cos\psi_{z}$. In order to quantify the degree of the line
shape asymmetry, we introduce a new parameter $D$, which is the ratio of the
asymmetric part over the symmetric part in the fitting function:

$\displaystyle~{}D(\delta
f)=\frac{\sum\limits_{i=1}^{2}\int_{\omega_{L}-2\pi\delta
f}^{\omega_{L}+2\pi\delta
f}\lvert\frac{d_{i}(\omega-\omega_{0})}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\rvert
d\omega}{\sum\limits_{i=1}^{2}\int_{\omega_{L}-2\pi\delta
f}^{\omega_{L}+2\pi\delta
f}\lvert\frac{c_{i}(\frac{\Gamma_{i}}{2})^{2}}{(\omega-\omega_{0})^{2}+(\frac{\Gamma_{i}}{2})^{2}}\rvert
d\omega}.$ (16)

The parameters in the equation above are the same as Eq. (14). Using the
fitting results, we have $D(3000)=2.38$ for the data at $\psi_{z}=30^{\circ}$
in Fig. 4 (b).

##  IV Solutions and discussions

The field-orientation dependent asymmetric magnetometer response, induced by
ac light shifts, presents a new source of heading error in Bell-Bloom
magnetometry. From Eqs. (8) and (13), we can get that the ac light shift
effect on the transmitted beam intensity is determined by $sB_{L1}$, the
product of the photon spin and the effective field from the light shift. It is
confirmed by the experiment that the asymmetry of the magnetometer response
can be reversed by flipping the pump beam polarization or detuning. Therefore,
to solve this heading error problem without consuming additional power, a
simple way is to reverse the pump beam polarization inside the atomic cell.

If the optical length of the pump beam inside the cell with two opposite
circular polarizations are same, the effect of
$\boldsymbol{s}\cdot\boldsymbol{P}^{\prime}_{x^{\prime}}$ in Eq. (6) can be
cancelled out. In the same way, the static light shift can also be spatially
averaged out. Therefore, the whole light shift effects in this magnetometry
can be eliminated using such a method. In practice, to maintain the
magnetometer signal, it is also important to reduce the possibility that atoms
polarized by one pump beam polarization diffuse into another region pumped by
light with an opposite polarization. Considering all the requirements, the
most suitable way to implement this scheme in the experiment is to attach a
half-wave plate in the middle of the cavity.

In the experiment, we use a customized true zero-order half-wave plate, which
is attached to a piece of K9 glass so that the total thickness of the wave
plate is 0.6 mm. Due to the heated environment around the cell, we need to
first check the temperature effect on the performance of these half-wave
plates. For a transverse wave propagating along the $z$ direction, the
oscillation amplitude of its electric field can be expressed as
$A_{x,y}=E_{x,y}\cos(\omega t+\delta_{x,y})$. The circular polarization part
in this wave can be characterized by an ellipticity parameter $\eta$ Born and
Wolf (1999):

$~{}\sin(2\eta)=\frac{2{E}_{x}{E}_{y}\sin(\delta_{x}-\delta_{y})}{|{E}_{x}|^{2}+|{E}_{y}|^{2}}.$
(17)

An ideally circularly polarized wave has $\eta=45^{\circ}$, and a linearly
polarized wave has $\eta=0^{\circ}$.

We attach one of the half-wave plates in the middle of the cavity as shown in
the inset of Fig. 5 (a), use a far-off-resonance light with $\eta$ close to
45∘ as the input beam, and measure $\eta$ of the output beam from the cavity
at different cell temperature. The experiment data in the same plot shows that
the output beam has a weak elliptical polarization at low temperature, due to
21 reflections on the cavity mirrors and 22 passes through the wave plate.
$\eta$ of the output beam changes slightly when the temperature is below 80
∘C, and the circular polarization of the output beam rapidly degrades when the
cell temperature is above 85 ∘C. Therefore, the customized half-wave plates
are feasible for applications when the cell temperature is below 80 ∘C, which
covers the sensor operation temperature in this paper.

Figure 5: (Color online) Plot (a) shows the ellipticity parameter $\eta$ of
the output beam as a function of the cell temperature, with the cavity
configuration shown in the inset. Plot(b) compares the magnetometer results
using two different cavity configurations. The lines are fitting curves using
Eq. (15).

Using this modified cavity with a half-wave plate in the middle, we retake the
magnetometer results at $\psi_{z}=30^{\circ}$. The new experiment data in Fig.
5 (b) shows $D^{\prime}(3000)=0.09$. Compared with the results using the
conventional cavity in the previous section, we have eliminated most of the
aforementioned light shift effect, and only $3.7\%$ of the original effect is
left. This residual effect can be further reduced by tuning the position of
the half-wave plate inside the cavity.

We use the parameter S/N as the magnetometer signal-to-noise ratio with a 1 Hz
bandwidth. Figure 6 (a) demonstrates that there are two orders of magnitude
difference for the maximum and minimum of S/N in the magnetometer using a
conventional cavity. To lift the detection dead zones in the sensor, we add a
reflection mirror to the conventional cavity, and arrange the cavity mirrors
as shown in the inset of Fig. 6 (a). In this new configuration, the optical
paths are bent by 90∘ in the middle. This path-bending cavity can be treated
as a combination of two separated normal multipass cavities. They can
complement each other because their dead zones are orthogonal.

Figure 6: (Color online) Plot (a) shows the magnetometer S/N as a function of
the $\psi_{z}$ using two cavity configurations with the pump beam resonant
with the Rb D1 line, $\omega=\omega_{L}$, and cell temperature around 55 ∘C.
The inset shows an illustration of the path-bending cavity. Plot (b) shows the
normalized response of the magnetometers taken at $\psi_{z}=45^{\circ}$, with
the pump beam 6 GHz blue detuned from the Rb D1 line. The lines are fitting
curves using Eq. (15), and the inset illustrates a path-bending cavity vapor
cell with half-wave plates.

While it helps to eliminate the detection dead zones, the introduction of an
extra reflection mirror also doubles the interactions between the mirror
surfaces and the beam, which can strongly influence the quality of the beam
circular polarization. In practice, we modify the cavity mirror parameters by
reducing the number of reflections inside the cavity to 13 times, and
increasing the length of each optical path inside the cavity to 26.3 mm. In
this way, the magnetometer signal amplitude is kept almost unchanged, and the
polarization of the beam inside the cavity is also maintained in a good
quality. As shown in Fig. 6 (a), S/N of the magnetometer using this path-
bending cavity fluctuates within 20% over the whole range of $\psi_{z}$. This
is a significant improvement compared with the magnetometer using a
conventional cavity, and the remaining orientation dependence of the S/N is
mainly determined by the cavity structure.

It is straightforward to combine the two methods developed above for
simultaneous elimination of the dead zones and light shifts. As shown in the
inset Fig. 6 (b), this is achieved by placing two half-wave plates in the
middle of the optical paths between the reflection mirror and cavity mirrors.
Its function is confirmed by the experiment results in the Fig. 6 (b), which
shows similar success on suppressing the light-shift effect using this new
configuration as the one in Fig. 5 (b).

##  V Conclusion

In summary, we have demonstrated an atomic orientation based scalar
magnetometer using a single amplitude-modulated beam and a multipass cell. We
suppress the light-shift effect by more than one order of magnitude, using a
half-wave plate in the middle of the cavity to reverse the beam polarization
inside the cavity. We also remove the detection dead zones, and limit the
change of the magnetometer signal-to-noise ratio within 20% over the whole
angular range, by modifying the cavity to a three-mirror configuration.
Combining these two techniques, we reached a light-shift-free and dead-zone-
free scalar magnetometer. The methods developed in this paper are novel in
optically pumped magnetometry, robust to use, and easy to be extended to other
atomic devices.

One important application is in the aforementioned atomic magnetometry using a
single elliptically polarized light Shah and Romalis (2009), where this beam
has to be kept off resonance for both optical pumping and Faraday rotation
detections. The inherent light shift problem limits the application of this
magnetometry in precise scalar field measurements. The polarization-reversing
cavity developed in this paper can solve this problem by eliminating the first
order light shift effect. Calculations based on Jones calculus have been
performed to confirm the feasibility of this scheme. If the proposed scheme
can be successfully implemented, this modified magnetometer would be a
promising compact single-beam version to replace the widely used two-beam
double-resonance scalar magnetometers Smullin  _et al._ (2009).

## Acknowledgements

This work was partially carried out at the USTC Center for Micro and Nanoscale
Research and Fabrication. This work was supported by National Natural Science
Foundation of China (Grant No. 11974329), and Scientific Instrument and
Equipment Development Projects, CAS (NO. YJKYYQ20200043).

## References

  * Smullin  _et al._ (2009) S. J. Smullin, I. M. Savukov, G. Vasilakis, R. K. Ghosh, and M. V. Romalis, Phys. Rev. A 80, 033420 (2009).
  * Sheng  _et al._ (2013) D. Sheng, S. Li, N. Dural, and M. V. Romalis, Phys. Rev. Lett. 110, 160802 (2013).
  * Lucivero  _et al._ (2021a) V. Lucivero, W. Lee, N. Dural, and M. Romalis, Phys. Rev. Applied 15, 014004 (2021a).
  * Lucivero  _et al._ (2021b) V. G. Lucivero, W. Lee, M. V. Romalis, M. E. Limes, E. L. Foley, and T. W. Kornack, (2021b), arXiv:2112.09004 [physics.atom-ph] .
  * Wilson  _et al._ (2019) N. Wilson, P. Light, A. Luiten, and C. Perrella, Phys. Rev. Applied 11, 044034 (2019).
  * Kitching (2018) J. Kitching, Appl. Phys. Rev. 5, 031302 (2018).
  * Abel  _et al._ (2020) C. Abel, S. Afach, N. J. Ayres, G. Ban, G. Bison, K. Bodek, V. Bondar, E. Chanel, P.-J. Chiu, C. B. Crawford, Z. Chowdhuri, M. Daum, S. Emmenegger, L. Ferraris-Bouchez, M. Fertl, B. Franke, W. C. Griffith, Z. D. Grujić, L. Hayen, V. Hélaine, N. Hild, M. Kasprzak, Y. Kermaidic, K. Kirch, P. Knowles, H.-C. Koch, S. Komposch, P. A. Koss, A. Kozela, J. Krempel, B. Lauss, T. Lefort, Y. Lemière, A. Leredde, A. Mtchedlishvili, P. Mohanmurthy, M. Musgrave, O. Naviliat-Cuncic, D. Pais, A. Pazgalev, F. M. Piegsa, E. Pierre, G. Pignol, P. N. Prashanth, G. Quéméner, M. Rawlik, D. Rebreyend, D. Ries, S. Roccia, D. Rozpedzik, P. Schmidt-Wellenburg, A. Schnabel, N. Severijns, R. T. Dinani, J. Thorne, A. Weis, E. Wursten, G. Wyszynski, J. Zejma, and G. Zsigmond, Phys. Rev. A 101, 053419 (2020).
  * Afach  _et al._ (2021) S. Afach  _et al._ (GNOME Collaboration), Nature Physics 17, 1396 (2021).
  * Prouty  _et al._ (2013) M. D. Prouty, R. Johnson, I. Hrvoic, and A. K. Vershovskiy, in  _Optical Magnetometry_ (Cambridge University Press, 2013) pp. 319–333.
  * Acuna (2002) M. H. Acuna, Rev. Sci. Instr. 73, 3717 (2002).
  * Limes  _et al._ (2020) M. Limes, E. Foley, T. Kornack, S. Caliga, S. McBride, A. Braun, W. Lee, V. Lucivero, and M. Romalis, Phys. Rev. Applied 14, 011002 (2020).
  * Zhang  _et al._ (2020) R. Zhang, W. Xiao, Y. Ding, Y. Feng, X. Peng, L. Shen, C. Sun, T. Wu, Y. Wu, Y. Yang, Z. Zheng, X. Zhang, J. Chen, and H. Guo, Science Advances 6, eaba8792 (2020).
  * Bloom (1962) A. L. Bloom, Appl. Opt. 1, 61 (1962).
  * B. Chéron and Hamel (2001) H. G. B. Chéron and J. Hamel, Eur. Phys. J. AP 13, 143 (2001).
  * Wang  _et al._ (2021) H. Wang, T. Wu, W. Xiao, H. Wang, X. Peng, and H. Guo, Phys. Rev. Applied 15, 024033 (2021).
  * Bao  _et al._ (2018) G. Bao, A. Wickenbrock, S. Rochester, W. Zhang, and D. Budker, Phys. Rev. Lett. 120, 033202 (2018).
  * Lee  _et al._ (2021) W. Lee, V. G. Lucivero, M. V. Romalis, M. E. Limes, E. L. Foley, and T. W. Kornack, Phys. Rev. A 103, 063103 (2021).
  * Oelsner  _et al._ (2019) G. Oelsner, V. Schultze, R. IJsselsteijn, F. Wittkämper, and R. Stolz, Phys. Rev. A 99, 013420 (2019).
  * Cohen-Tannoudji (1962) C. Cohen-Tannoudji, Ann. Phys. (Paris) 13, 423 (1962).
  * Happer (1972) W. Happer, Rev. Mod. Phys. 44, 169 (1972).
  * Norris  _et al._ (2012) D. G. Norris, A. D. Cimmarusti, L. A. Orozco, P. Barberis-Blostein, and H. J. Carmichael, Phys. Rev. A 85, 021804 (2012).
  * Mathur  _et al._ (1968) B. S. Mathur, H. Tang, and W. Happer, Phys. Rev. 171, 11 (1968).
  * Appelt  _et al._ (1998) S. Appelt, A. B. Baranga, C. J. Erickson, M. Romalis, A. R. Young, and W. Happer, Phys. Rev. A 58, 1412 (1998).
  * Zhang  _et al._ (2022) R. Zhang, D. Kanta, A. Wickenbrock, H. Guo, and D. Budker,  (2022), arXiv:2204.05071 [physics.atom-ph] .
  * Shah and Romalis (2009) V. Shah and M. V. Romalis, Phys. Rev. A 80, 013416 (2009).
  * Ben-Kish and Romalis (2010) A. Ben-Kish and M. V. Romalis, Phys. Rev. Lett. 105, 193601 (2010).
  * Cai  _et al._ (2020) B. Cai, C.-P. Hao, Z.-R. Qiu, Q.-Q. Yu, W. Xiao, and D. Sheng, Phys. Rev. A 101, 053436 (2020).
  * Hao  _et al._ (2021) C.-P. Hao, Q.-Q. Yu, C.-Q. Yuan, S.-Q. Liu, and D. Sheng, Phys. Rev. A 103, 053523 (2021).
  * Li  _et al._ (2011) S. Li, P. Vachaspati, D. Sheng, N. Dural, and M. V. Romalis, Phys. Rev. A 84, 061403 (2011).
  * Bell and Bloom (1961) W. E. Bell and A. L. Bloom, Phys. Rev. Lett. 6, 280 (1961).
  * Appelt  _et al._ (1999) S. Appelt, A. Ben-Amar Baranga, A. R. Young, and W. Happer, Phys. Rev. A 59, 2078 (1999).
  * Savukov and Romalis (2005) I. M. Savukov and M. V. Romalis, Phys. Rev. A 71, 023405 (2005).
  * Kornack (2005) T. W. Kornack, Ph.d. thesis, Princeton University (2005).
  * Xiao  _et al._ (2006) Y. Xiao, I. Novikova, D. F. Phillips, and R. L. Walsworth, Phys. Rev. Lett. 96, 043601 (2006).
  * Lucivero  _et al._ (2017) V. G. Lucivero, N. D. McDonough, N. Dural, and M. V. Romalis, Phys. Rev. A 96, 062702 (2017).
  * Seltzer (2008) S. J. Seltzer, Ph.d. thesis, Princeton University (2008).
  * Born and Wolf (1999) M. Born and E. Wolf, in  _Principles of Optics_ (Cambridge University Press, 1999) pp. 25–33, 7th ed.