# Contrastive Example-Based Control

Kyle Hatch 1<EMAIL_ADDRESS>
Benjamin Eysenbach 2<EMAIL_ADDRESS>
Rafael Rafailov 1<EMAIL_ADDRESS>
Tianhe Yu 1<EMAIL_ADDRESS>
Ruslan Salakhutdinov 2<EMAIL_ADDRESS>
Sergey Levine 3<EMAIL_ADDRESS>
Chelsea Finn 1<EMAIL_ADDRESS>
1Department of Computer Science  Stanford University
2Machine Learning Department  Carnegie Mellon University
3Department of Electrical Engineering and Computer Sciences  UC Berkeley

###### Abstract

While many real-world problems that might benefit from reinforcement learning,
these problems rarely fit into the MDP mold: interacting with the environment
is often expensive and specifying reward functions is challenging. Motivated
by these challenges, prior work has developed data-driven approaches that
learn entirely from samples from the transition dynamics and examples of high-
return states. These methods typically learn a reward function from high-
return states, use that reward function to label the transitions, and then
apply an offline RL algorithm to these transitions. While these methods can
achieve good results on many tasks, they can be complex, often requiring
regularization and temporal difference updates. In this paper, we propose a
method for offline, example-based control that learns an implicit model of
multi-step transitions, rather than a reward function. We show that this
implicit model can represent the Q-values for the example-based control
problem. Across a range of state-based and image-based offline control tasks,
our method outperforms baselines that use learned reward functions; additional
experiments demonstrate improved robustness and scaling with dataset
size.111Videos of our method are available on the project website:
https://sites.google.com/view/laeo-rl. Code is released at:
https://github.com/khatch31/laeo.

###### keywords:

reinforcement learning, offline RL, robot learning, reward learning,
contrastive learning, model-based reinforcement learning, example-based
control, reward-free learning

##  1 Introduction

Reinforcement learning is typically framed as the problem of maximizing a
given reward function. However, in many real-world situations, it is more
natural for users to define what they want an agent to do with examples of
successful outcomes (Fu et al., 2018b; Zolna et al., 2020a; Xu and Denil,
2019; Eysenbach et al., 2021). For example, a user that wants their robot to
pack laundry into a washing machine might provide multiple examples of states
where the laundry has been packed correctly. This problem setting is often
seen as a variant of inverse reinforcement learning (Fu et al., 2018b), where
the aim is to learn only from examples of successful outcomes, rather than
from expert demonstrations. To solve this problem, the agent must both figure
out what constitutes task success (i.e., what the examples have in common) and
how to achieve such successful outcomes.

In this paper, our aim is to address this problem setting in the case where
the agent must learn from offline data without trial and error. Instead, the
agent must infer the outcomes of potential actions from the provided data,
while also relating these inferred outcomes to the success examples. We will
refer to this problem of offline RL with success examples as _offline example-
based control_.

Most prior approaches involve two steps: _first_ learning a reward function,
and _second_ combining it with an RL method to recover a policy (Fu et al.,
2018b; Zolna et al., 2020a; Xu and Denil, 2019). While such approaches can
achieve excellent results when provided sufficient data (Kalashnikov et al.,
2021; Zolna et al., 2020a), learning the reward function is challenging when
the number of success examples is small (Li et al., 2021; Zolna et al.,
2020a). Moreover, these prior approaches are relatively complex (e.g., they
use temporal difference learning) and have many hyperparameters.

Our aim is to provide a simple and scalable approach that avoids the
challenges of reward learning. The main idea will be learning a certain type
of dynamics model. Then, using that model to predict the probabilities of
reaching each of the success examples, we will be able to estimate the
Q-values for every state and action. Note that this approach does not use an
offline RL algorithm as a subroutine. The key design decision is the model
type; we will use an implicit model of the time-averaged future (precisely,
the discounted state occupancy measure). This decision means that our model
reasons across multiple time steps but will not output high-dimensional
observations (only a scalar number). A limitation of this approach is that it
will correspond to a single step of policy improvement: the dynamics model
corresponds to the dynamics of the behavioral policy, not of the reward-
maximizing policy. While this means that our method is not guaranteed to yield
the optimal policy, our experiments nevertheless show that our approach
outperforms multi-step RL methods.

The main contribution of this paper is an offline RL method (LAEO) that learns
a policy from examples of high-reward states. The key idea behind LAEO is an
implicit dynamics model, which represents the probability of reaching states
at some point in the future. We use this model to estimate the probability of
reaching examples of high-return states. LAEO is simpler yet more effective
than prior approaches based on reward classifiers. Our experiments demonstrate
that LAEO can successfully solve offline RL problems from examples of high-
return states on four state-based and two image-based manipulation tasks. Our
experiments show that LAEO is more robust to occlusions and also exhibits
better scaling with dataset size than prior methods. We show that LAEO can
work in example-based control settings in which goal-conditioned RL methods
fail. Additionally, we show that the dynamics model learned by LAEO can
generalize to multiple different tasks, being used to solve tasks that are not
explicitly represented in the training data.

##  2 Related Work

#### Reward learning.

To overcome the challenge of hand-engineering reward functions for RL, prior
methods either use supervised learning or adversarial training to learn a
policy that matches the expert behavior given by the demonstration (imitation
learning) (Pomerleau, 1988; Ross et al., 2011; Ho and Ermon, 2016; Spencer et
al., 2021) or learn a reward function from demonstrations and optimize the
policy with the learned reward through trial and error (inverse RL) (Ng and
Russell, 2000; Abbeel and Ng, 2004; Ratliff et al., 2006; Ziebart et al.,
2008; Finn et al., 2016; Fu et al., 2018a). However, providing full
demonstrations complete with agent actions is often difficult, therefore,
recent works have focused on the setting where only a set of user-specified
goal states or human videos are available (Fu et al., 2018b; Singh et al.,
2019; Kalashnikov et al., 2021; Xie et al., 2018; Eysenbach et al., 2021; Chen
et al., 2021). These reward learning approaches have shown successes in real-
world robotic manipulation tasks from high-dimensional imageinputs (Finn et
al., 2016; Singh et al., 2019; Zhu et al., 2020; Chen et al., 2021).
Nevertheless, to combat covariate shift that could lead the policy to drift
away from the expert distribution, these methods usually require significant
online interaction. Unlike these works that study online settings, we consider
learning visuomotor skills from offline datasets.

#### Offline RL.

Offline RL (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012; Levine
et al., 2020) studies the problem of learning a policy from a static dataset
without online data collection in the environment, which has shown promising
results in robotic manipulation (Kalashnikov et al., 2018; Mandlekar et al.,
2020; Rafailov et al., 2021; Singh et al., 2020; Julian et al., 2020;
Kalashnikov et al., 2021). Prior offline RL methods focus on the challenge of
distribution shift between the offline training data and deployment using a
variety of techniques, such as policy constraints (Fujimoto et al., 2018; Liu
et al., 2020; Jaques et al., 2019; Wu et al., 2019; Zhou et al., 2020; Kumar
et al., 2019; Siegel et al., 2020; Peng et al., 2019; Fujimoto and Gu, 2021;
Ghasemipour et al., 2021), conservative Q-functions (Kumar et al., 2020;
Kostrikov et al., 2021; Yu et al., 2021; Sinha and Garg, 2021), and penalizing
out-of-distribution states generated by learned dynamics models (Kidambi et
al., 2020; Yu et al., 2020b; Matsushima et al., 2020; Argenson and Dulac-
Arnold, 2020; Swazinna et al., 2020; Rafailov et al., 2021; Lee et al., 2021;
Yu et al., 2021).

While these prior works successfully address the issue of distribution shift,
they still require reward annotations for the offline data. Practical
approaches have used manual reward sketching to train a reward model (Cabi et
al., 2019; Konyushkova et al., 2020; Rafailov et al., 2021) or heuristic
reward functions (Yu et al., 2022). Others have considered offline learning
from demonstrations, without access to a predefined reward function (Mandlekar
et al., 2020; Zolna et al., 2020a; Xu et al., 2022; Jarboui and Perchet,
2021), however they rely on high-quality demonstration data. In contrast, our
method: _(1)_ addresses distributional shift induced by both the learned
policy and the reward function in a principled way, _(2)_ only requires user-
provided goal states and _(3)_ does not require expert-quality data, resulting
in an effective and practical offline reward learning scheme.

##  3 Learning to Achieve Examples Offline

Offline RL methods typically require regularization, and our method will
employ regularization in two ways. First, we regularize the policy with an
additional behavioral cloning term, which penalizes the policy for sampling
out-of-distribution actions. Second, our method uses the Q-function for the
behavioral policy, so it performs one (not many) step of policy improvement.
These regularizers mean that our approach is not guaranteed to yield the
optimal policy.

###  3.1 Preliminaries

We assume that an agent interacts with an MDP with states $s\in{\mathcal{S}}$,
actions $a$, a state-only reward function $r(s)\geq 0$, initial state
distribution $p_{0}(s_{0})$ and dynamics $p(s_{t+1}\mid s_{t},a_{t})$. We use
$\tau=(s_{0},a_{0},s_{1},a_{1},\cdots)$ to denote an infinite-length
trajectory. The likelihood of a trajectory under a policy $\pi(a\mid s)$ is
$\pi(\tau)=p_{0}(s_{0})\prod_{t=0}^{\infty}p(s_{t+1}\mid
s_{t},a_{t})\pi(a_{t}\mid s_{t})$. The objective is to learn a policy
$\pi(a\mid s)$ that maximizes the expected, $\gamma$-discounted sum of
rewards:
$\max_{\pi}\mathbb{E}_{\pi(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t})\right].$
We define the Q-function for policy $\pi$ as the expected discounted sum of
returns, conditioned on an initial state and action:

$Q^{\pi}(s,a)\triangleq\mathbb{E}_{\pi(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t})\bigg{|}\begin{subarray}{c}s_{0}=s\\\
a_{0}=a\end{subarray}\right].$ (1)

We will focus on the offline (i.e., batch RL) setting. Instead of learning by
interacting with the environment (i.e., via trial and error), the RL agent
will receive as input a dataset of trajectories
${\mathcal{D}}_{\tau}=\\{\tau\sim\beta(\tau)\\}$ collected by a behavioral
policy $\beta(a\mid s)$. We will use $Q^{\beta}(s,a)$ to denote the Q-function
of the behavioral policy.

#### Specifying the reward function.

In many real-world applications, specifying and measuring a scalar reward
function is challenging, but providing examples of good states (i.e., those
which would receive high rewards) is straightforward. Thus, we follow prior
work (Fu et al., 2018b; Zolna et al., 2020a; Eysenbach et al., 2021; Xu and
Denil, 2019; Zolna et al., 2020b) in assuming that the agent does not observe
scalar rewards (i.e., ${\mathcal{D}}_{\tau}$ does not contain reward
information). Instead, the agent receives as input a dataset
${\mathcal{D}}_{*}=\\{s^{*}\\}$ of high-reward states $s^{*}\in{\mathcal{S}}$.
These high-reward states are examples of good outcomes, which the agent would
like to achieve. The high-reward states are not labeled with their specific
reward value.

To make the control problem well defined, we must relate these success
examples to the reward function. We do this by assuming that the frequency of
each success example is proportional to its reward: good states are more
likely to appear (and be duplicated) as success examples.

###### Assumption 1

Let $p_{\tau}(s)$ be the empirical probability density of state $s$ in the
trajectory dataset, and let $p_{*}(s)$ as the empirical probability density of
state $s$ under the high-reward state dataset. We assume that there exists a
positive constant $c$ such that $r(s)=c\frac{p_{*}(s)}{p_{\tau}(s)}$ for all
states $s$.

This is the same assumption as Eysenbach et al. (2021). This assumption is
important because it shows how example-based control is universal: for any
reward function, we can specify the corresponding example-based problem by
constructing a dataset of success examples that are sampled according to their
rewards. We assumed that rewards are non-negative so that these sampling
probabilities are positive.

This assumption can also be read in reverse. When a user constructs a dataset
of success examples in an arbitrary fashion, they are implicitly defining a
reward function. In the tabular setting, the (implicit) reward function for
state $s$ is the count of the times $s$ occurs in the dataset of success
examples. Compared with goal-conditioned RL (Kaelbling, 1993), defining tasks
via success examples is more general. By identifying what all the success
examples have in common (e.g., laundry is folded), the RL agent can learn what
is necessary to solve the task and what is irrelevant (e.g., the color of the
clothes in the laundry). We now can define our problem statement as follows:

######  Definition 3.1.

In the offline example-based control problem, a learning algorithm receives as
input a dataset of trajectories ${\mathcal{D}}_{\tau}=\\{\tau\\}$ and a
dataset of successful outcomes ${\mathcal{D}}_{*}=\\{s\\}$ satisfying
Assumption 1. The aim is to output a policy that maximizes the RL objective
(Eq. 3.1).

This problem setting is appealing because it mirrors many practical RL
applications: a user has access to historical data from past experience, but
collecting new experience is prohibitively expensive. Moreover, this problem
setting can mitigate the challenges of reward function design. Rather than
having to implement a reward function and add instruments to measure the
corresponding components, the users need only provide a handful of
observations that solved the task. This problem setting is similar to
imitation learning, in the sense that the only inputs are data. However,
unlike imitation learning, in this problem setting the high-reward states are
not labeled with actions, and these high-reward states may not necessarily
contain entire trajectories.

Figure 1: Our method will use contrastive learning to predict which states
might occur at some point in the future.

Our method will estimate the discounted state occupancy measure,

$\displaystyle p^{\beta}(s_{t+}=s\mid s_{0},a_{0})$
$\displaystyle\triangleq(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}p_{t}^{\pi}(s_{t}=s\mid
s_{0},a_{0}),$ (2)

where $p_{t}^{\beta}(s_{t}\mid s,a)$ is the probability of policy $\beta(a\mid
s)$ visiting state $s_{t}$ after exactly $t$ time steps. Unlike the transition
function $p(s_{t+1}\mid s_{t},a_{t})$, the discounted state occupancy measure
indicates the probability of visiting a state at any point in the future, not
just at the immediate next time step. In tabular settings, this distribution
corresponds to the successor representations (Dayan, 1993). To handle
continuous settings, we will use the contrastive approach from recent work
(Mazoure et al., 2020; Eysenbach et al., 2022). We will learn a function
$f(s,a,s_{f})\in\mathbbm{R}$ takes as input an initial state-action pair as
well as a candidate future state, and outputs a score estimating the
likelihood that $s_{f}$ is a real future state. The loss function is a
standard contrastive learning loss(e.g., Ma and Collins (2018)), where
positive examples are triplets of a state, action, and future state:

$\max_{f}{\mathcal{L}}(f;{\mathcal{D}}_{\tau})\triangleq\mathbb{E}_{p(s,a),s_{f}\sim
p^{\beta}(s_{t+}\mid
s,a)}\left[\log\sigma(f(s,a,s_{f}))\right]+\mathbb{E}_{p(s,a),s_{f}\sim
p(s)}\left[\log(1-\sigma(f(s,a,s_{f})))\right],$

where $\sigma(\cdot)$ is the sigmoid function. At optimality, the implicit
dynamics model encodes the discounted state occupancy measure:

$f^{*}(s,a,s_{f})=\log p^{\beta}(s_{t+}=s_{f}\mid s,a)-\log p_{\tau}(s_{f}).$
(3)

We visualize this implicit dynamics model in Fig. 1. Note that this dynamics
model is policy dependent. Because it is trained with data collected from one
policy ($\beta(a\mid s)$), it will correspond to the probability that _that_
policy visits states in the future. Because of this, our method will result in
estimating the value function for the behavioral policy (akin to 1-step RL
(Brandfonbrener et al., 2021)), and will not perform multiple steps of policy
improvement. Intuitively, the training of this implicit model resembles
hindsight relabeling (Kaelbling, 1993; Andrychowicz et al., 2017). However, it
is generally unclear how to use hindsight relabeling for single-task problems.
Despite being a single-task method, our method will be able to make use of
hindsight relabeling to train the dynamics model.

###  3.2 Deriving Our Method

The key idea behind out method is that this implicit dynamics model can be
used to represent the Q-values for the example-based problem, up to a
constant. The proof is in Appendix A.

######  Lemma 3.2.

Assume that the implicit dynamics model is learned without errors. Then the
Q-function for the data collection policy $\beta(a\mid s)$ can be expressed in
terms of this implicit dynamics model:

$\displaystyle Q^{\beta}(s,a)$
$\displaystyle=\frac{c}{1-\gamma}\mathbb{E}_{p_{*}(s^{*})}\left[e^{f(s,a,s^{*})}\right].$
(4)

So, after learning the implicit dynamics model, we can estimate the Q-values
by averaging this model’s predictions across the success examples. We will
update the policy using Q-values estimated in this manner, plus a
regularization term:

$\min_{\pi}{\mathcal{L}}(\pi;f,{\mathcal{D}}_{*})\triangleq-(1-\lambda)\mathbb{E}_{\pi(a\mid
s)p(s),s^{*}\sim{\mathcal{D}}_{*}}\left[e^{f(s,a,s^{*})}\right]-\lambda\mathbb{E}_{s,a\sim{\mathcal{D}}_{\tau}}\left[\log\pi(a\mid
s)\right].$ (5)

In our experiments, we use a weak regularization coefficient of $\lambda=0.5$.

Figure 2: If the state-action representation $\phi(s,a)$ is close to the
representation of a high-return state $\psi(s)$, then the policy is likely to
visit that state. Our method estimates Q-values by combining the distances to
all the high-return states (Eq. 1).

It is worth comparing this approach to prior methods based on learned reward
functions (Xu and Denil, 2019; Fu et al., 2018b; Zolna et al., 2020a). Those
methods learn a reward function from the success examples, and use that
learned reward function to synthetically label the dataset of trajectories.
Both approaches can be interpreted as learning a function on one of the
datasets and then applying that function to the other dataset. Because it is
easier to fit a function when given large quantities of data, we predict that
our approach will outperform the learned reward function approach when the
number of success examples is small, relative to the number of unlabeled
trajectories. Other prior methods (Eysenbach et al., 2021; Reddy et al., 2020)
avoid learning reward functions by proposing TD update rules that are applied
to both the unlabeled transitions and the high-return states. However, because
these methods have yet to be adapted to the offline RL setting, we will focus
our comparisons on the reward-learning methods.

###  3.3 A Geometric Perspective

Before presenting the complete RL algorithm, we provide a geometric
perspective on the representations learned by our method. Our implicit models
learns a representation of state-action pairs $\phi(s,a)$ as well as a
representation of future states $\psi(s)$. One way that our method can
optimize these representations is by treating $\phi(s,a)$ as a prediction for
the future representations.222Our method can also learn the opposite, where
$\psi(s)$ is a prediction for the previous representations. Each of the high-
return states can be mapped to the same representation space. To determine
whether a state-action pair has a large or small Q-value, we can simply see
whether the predicted representation $\phi(s,a)$ is close to the
representations of any of the success examples. Our method learns these
representations so that the Q-values are directly related to the Euclidean
distances333When representations are normalized, the dot product is equivalent
to the Euclidean norm. We find that unnormalized features work better in our
experiments. from each success example. Thus, our method can be interpreted as
learning a representation space such that estimating Q-values corresponds to
simple geometric operations (kernel smoothing with an RBF kernel (Hastie et
al., 2009, Chpt. 6)) on the learned representations. While the example-based
control problem is more general than goal-conditioned RL (see Sec. 3.1), we
can recover goal-conditioned RL as a special case by using a single success
example.

###  3.4 A Complete Algorithm

We now build a complete offline RL algorithm based on these Q-functions. We
will call our method Learning to Achieve Examples Offline (LAEO). Our
algorithm will resemble one-step RL methods, but differ in how the Q-function
is trained. After learning the implicit dynamics model (and, hence,
Q-function) we will optimize the policy. The objective for the policy is
maximizing (log) Q-values plus a regularization term, which penalizes sampling
unseen actions:444For all experiments except Fig. 8, we apply Jensen’s
inequality to the first term, using $\mathbb{E}_{\pi(a\mid s),s^{*}\sim
p_{*}(s)}[f(s,a,s^{*})]$.

$\displaystyle\max_{\pi}\;$ $\displaystyle(1-\lambda)\log\mathbb{E}_{\pi(a\mid
s)p_{\tau}(s)}\left[Q(s,a)\right]+\lambda\mathbb{E}_{(s,a)\sim
p_{\tau}(s,a)}\left[\log\pi(a\mid s)\right]$
$\displaystyle=(1-\lambda)\log\mathbb{E}_{\pi(a\mid s),s^{*}\sim
p_{*}(s)}\left[e^{f(s,a,s^{*})}\right]+\lambda\mathbb{E}_{(s,a)\sim
p_{\tau}(s,a)}\left[\log\pi(a\mid s)\right].$ (6)

Algorithm 1 Learning to Achieve Examples Offline

1:Inputs: dataset of trajectories ${\mathcal{D}}=\\{\tau\\}$, …………. dataset of
high-return states ${\mathcal{D}}_{*}=\\{s\\}$.

2:Learn the model via contrastive learning: ……………
$f\leftarrow\operatorname*{arg\,min}_{f}{\mathcal{L}}(f;{\mathcal{D}}_{\tau})$
$\triangleright$ Eq. 5

3:Learn the policy:
$\pi\leftarrow\operatorname*{arg\,min}_{\pi}{\mathcal{L}}(\pi;f,{\mathcal{D}}_{*})$
$\triangleright$ Eq. 6

4:return policy $\pi(a\mid s)$

As noted above, this is a one-step RL method: it updates the policy to
maximize the Q-values of the behavioral policy. Performing just a single step
of policy improvement can be viewed as a form of regularization in RL, in the
same spirit as early stopping is a form of regularization in supervised
learning. Prior work has found that one-step RL methods can perform well in
the offline RL setting. Because our method performs only a single step of
policy improvement, we are not guaranteed that it will converge to the reward-
maximizing policy. We summarize the complete algorithm in Alg. 1.

##  4 Experiments

Figure 3: Benchmark tasks: We evaluate the performance of LAEO on six
simulated manipulation tasks, two of which use pixel observations (FetchReach-
image and FetchPush-image) and four of which use low-dimensional states
(FetchReach, FetchPush, SawyerWindowOpen, and SawyerDrawerClose ).

Our experiments test whether LAEO can effectively solve offline RL tasks that
are specified by examples of high-return states, rather than via scalar reward
functions. We study when our approach outperforms prior approaches based on
learned reward functions. We look not only at the performance relative to
baselines on state-based and image-based tasks, but also how that performance
depends on the size and composition of the input datasets. Additional
experiments study how LAEO performs when provided with varying numbers of
success observations and whether our method can solve partially observed
tasks. We include full hyperparameters and implementation details in Appendix
B. Code is available at https://github.com/khatch31/laeo. Videos of our method
are available at https://sites.google.com/view/laeo-rl.

#### Baselines.

Our main point of comparison will be prior methods that use learned reward
functions: ORIL (Zolna et al., 2020a) and PURL (Xu and Denil, 2019). The main
difference between these methods is the loss function used to train reward
function: ORIL uses binary cross entropy loss while PURL uses a positive-
unlabeled loss (Xu and Denil, 2019). Note that the ORIL paper also reports
results using a positive-unlabeled loss, but for the sake of clarity we simply
refer to it as PURL. After learning the reward function, each of these methods
applies an off-the-shelf RL algorithm. We will implement all baselines using
the TD3+BC (Fujimoto and Gu, 2021) offline RL algorithm. These offline RL
methods achieve good performance on tasks specified via reward functions
(Kostrikov et al., 2021; Brandfonbrener et al., 2021; Fujimoto and Gu, 2021).
We also include Behavioral Cloning (BC) results.

Figure 4: Benchmark comparison: LAEO matches or outperforms prior example-
based offline RL methods on state and image-based tasks, including those that
learn a separate reward function (ORIL, PURL).The gap in performance is most
significant on the FetchPush and FetchPush-image tasks, which involve more
complicated dynamics than the other tasks, suggesting that LAEO may outperform
model free reward-learning approaches on tasks with complicated dynamics. LAEO
also outperforms BC on all of the tasks, highlighting LAEO’s ability to learn
a policy that outperforms the behavior policy on non-demonstration datasets.

#### Benchmark comparison.

We start by comparing the performance of LAEO to these baselines on six
manipulation tasks. FetchReach and FetchPush are two manipulation tasks from
Plappert et al. (2018) that use state-based observations. FetchReach-image and
FetchPush-image are the same tasks but with image-based observations.
SawyerWindowOpen and Sawyer- DrawerClose are two manipulation tasks from Yu et
al. (2020a). For each of these tasks, we collect a dataset of medium quality
by training an online agent from Eysenbach et al. (2022) and rolling out
multiple checkpoints during the course of training. The resulting datasets
have success rates between $45\%-50\%$. We report results after $500,000$
training gradient steps (or $250,000$ steps, if the task success rates have
converged by that point).

We report results in Fig. 4. We observe that LAEO, PURL, and ORIL perform
similarly on FetchReach and FetchReach-image. This is likely because these are
relatively easy tasks, and each of these methods is able to achieve a high
success rate. Note that all of these methods significantly outperform BC,
indicating that they are able to learn better policies than the mode behavior
policies represented in the datasets. On SawyerDrawerClose, all methods,
including BC, achieve near perfect success rates, likely due to the simplicity
of this task. On FetchPush, FetchPush-image, and SawyerWindowOpen, LAEO
outperforms all of the baselines by a significant margin. Recall that the main
difference between LAEO and PURL/ORIL is by learning a dynamics model, rather
than the reward function. These experiments suggest that for tasks with more
complex dynamics, learning a dynamics model can achieve better performance
than is achieved by model-free reward classifier methods.

Figure 5: Data quality. LAEO continues to match or outperform reward
classifier based methods on datasets that contain a low percentage of
successful trajectories.

#### Varying the input data.

Our next experiment studies how the dataset composition affects LAEO and the
baselines. On each of three tasks, we generate a low-quality dataset by
rolling out multiple checkpoints from a partially trained agent from Eysenbach
et al. (2022). In comparison to the medium-quality datasets collected earlier,
which have success rates between $45\%-50\%$, these low quality datasets have
success rates between $8\%-12\%$. We will denote these low quality datasets
with the “Hard” suffix. Fig. 5 shows that LAEO continues to outperform
baselines on these lower-quality datasets.

Figure 6: Effect of dataset size: (Left) The most competitive baseline (ORIL)
achieves better performance when given more examples of high-return states,
likely because it makes it easier to learn ORIL’s reward classifier. LAEO,
which does not require learning a reward classifier, consistently achieves
high success rates. (Right) LAEO continues to improve when trained with more
reward-free trajectories, while ORIL’s performance plateaus.

Our next experiments study how varying the number of high-return example
states and the number of reward-free trajectories affects performance. As
noted in the Sec. 1, we conjecture that our method will be especially
beneficial relative to reward-learning approaches in settings with very few
high-return example states. In Fig. 6 _(left)_ , we vary the number of high-
return example states on FetchPush -image, holding the number of unlabeled
trajectories constant. We observe that LAEO maintains achieves the same
performance with 1 success example as with 200 success examples. In contrast,
ORIL’s performance decreases as the number of high-return example states
decreases. In Fig. 6 _(right)_ , we vary the number of unlabeled trajectories,
holding the number of high-return example states constant at $200$. We test
the performance of LAEO vs. ORIL on three different dataset sizes on
FetchPush-image, roughly corresponding to three different orders of magnitude:
the $0.1\times$ dataset contains $3,966$ trajectories, the $1\times$ dataset
contains $31,271$ trajectories, and the $10\times$ dataset contains $300,578$
trajectories. We observe that LAEO continues to see performance gains as
number of unlabeled trajectories increases, whereas ORIL’s performance
plateaus. Taken together these results suggest that, in comparison to reward
classifier based methods, LAEO needs less human supervision and is more
effective at leveraging large quantities of unlabeled data.

#### Partial Observability.

Figure 7: Partial observability. LAEO continues to solve the FetchPush-image
manipulation task in a setting where the new camera placement causes partial
observability. This camera angle causes the block to be hidden from view by
the gripper when the gripper reaches down to push the block.

We also test the performance of LAEO on a partially-observed task. We modify
the camera position in the FetchPush-image so that the block is occluded
whenever the end effector is moved to touch the block. While such partial
observability can stymie temporal difference methods (Whitehead and Ballard,
1991), we predict that LAEO might continue to solve this task because it does
not rely on temporal difference learning. The results, shown in Fig. 7,
confirm this prediction. On this partially observable task, we compare the
performance of LAEO with that of ORIL, the best performing baseline on the
fully observable tasks. On the partially observable task, LAEO achieves a
success rate of $51.9\%$, versus $33.9\%$ for ORIL.

Figure 8: Comparison with goal-conditioned RL. LAEO solves manipulation tasks
at multiple different locations without being provided with a goal-state at
test time.

#### Comparison to Goal-Conditioned RL.

One of the key advantages of example-based control, relative to goal-
conditioned RL, is that the policy can identify common patterns in the success
examples to solve tasks in scenarios where it has never before seen a success
example. In settings such as robotics, this can be an issue since acquiring a
goal state to provide to the agent requires already solving the desired task
in the first place. We test this capability in a variant of the
SawyerDrawerClose environment. For training, the drawer’s X position is chosen
as one of five fixed locations. Then, we evaluate the policy learned by LAEO
on three types of environments: _In Distribution_ : the drawer’s X position is
one of the five locations from training; _Interpolation_ : The drawer’s X
position is between some of the locations seen during training;
_Extrapolation_ : The drawer’s X position is outside the range of X positions
seen during training. We compare to a goal-conditioned policy learned via
contrastive RL, where actions are extracted by averaging over the (training)
success examples: $\pi(a\mid s)=\mathbb{E}_{s^{*}\sim p_{*}(s)}[\pi(a\mid
s,g=s^{*})]$.

The results, shown in Fig. 8, show that LAEO consistently outperforms this
goal-conditioned baseline. As expected, the performance is highest for the In
Distribution environments and lowest for the Extrapolation environments. Taken
together, these experiments show that LAEO can learn to reach multiple
different goal locations without access to goal states during test time.

#### Multitask Critic.

Figure 9: Multitask Critic: Cross entropy method (CEM) optimization over the
LAEO dynamics model trained only on the data from the drawer close task is
able to solve six different tasks. Randomly sampling actions from the action
space results in a $0\%$ success rate across all of the six tasks (not shown
for clarity).

We explore whether a LAEO dynamics network trained on data from one task can
be used to solve other downstream tasks. We create a simple multitask
environment by defining several different tasks that can be solved in the
SawyerDrawerClose environment: Close, Half-closed, Open, Reach-near, Reach-
medium, and Reach-far. We then use a trained critic network from the previous
set of experiments (Comparison to Goal-Conditioned RL), condition it on a
success example from a downstream task, and select actions by using cross
entropy method (CEM) optimization. By using CEM optimization, we do not need
to train a separate policy network for each of the tasks. See Appendix C for
implementation details and for details of the multitask drawer environment.

CEM over the LAEO critic achieves non-zero success rates on all six tasks,
despite only being trained on data from the Close task (see Figure 9). In
contrast, randomly sampling actions from the action space achieves a $0\%$
success rate on all of the tasks. Results are averaged across eight random
seeds. This suggests that a single LAEO critic can be leveraged to solve
multiple downstream tasks, as long as the dynamics required to solve those
tasks are represented in the training data. Note that since we condition the
critic network on a single goal example, these experiments can be interpreted
from a goal-conditioned perspective as well as an example-based control
perspective. In future work, we aim to explore the multitask capabilities of
the LAEO dynamics model in an example-based control setting at a larger scale.
This will involve training on larger, more diverse datasets as well as
conditioning the critic network on multiple success examples for a single task
(as done in the Comparison to Goal-Conditioned RL experiments).

##  5 Conclusion

In this paper, we present an RL algorithm aimed at settings where data
collection and reward specification are difficult. Our method learns from a
combination of high-return states and reward-free trajectories, integrating
these two types of information to learn reward-maximizing policies. Whereas
prior methods perform this integration by learning a reward function and then
applying an off-the-shelf RL algorithm, ours learns an implicit dynamics
model. Not only is our method simpler (no additional RL algorithm required!),
but also it achieves higher success rates than prior methods.

While our experiments only start to study the ability of contrastive-based
methods to scale to high-dimensional observations, we conjecture that methods
like LAEO may be particularly amenable to such problems because the method for
learning the representations (contrastive learning) resembles prior
representation learning methods (Mazoure et al., 2020; Nair et al., 2022).
Scaling this method to very large offline datasets is an important direction
for future work.

##  6 Acknowledgments

BE is supported by the Fannie and John Hertz Foundation and the NSF GRFP
(DGE2140739).

## References

  * Abbeel and Ng (2004) Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 1, 2004.
  * Andrychowicz et al. (2017) Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  _arXiv preprint arXiv:1707.01495_ , 2017.
  * Argenson and Dulac-Arnold (2020) Arthur Argenson and Gabriel Dulac-Arnold.  Model-based offline planning.  _arXiv preprint arXiv:2008.05556_ , 2020.
  * Brandfonbrener et al. (2021) David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna.  Offline rl without off-policy evaluation.  _Advances in Neural Information Processing Systems_ , 34:4933–4946, 2021.
  * Cabi et al. (2019) Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zołna, Yusuf Aytar, David Budden, Mel Vecerik, et al.  A framework for data-driven robotics.  _arXiv preprint arXiv:1909.12200_ , 2019.
  * Chen et al. (2021) Annie S Chen, Suraj Nair, and Chelsea Finn.  Learning generalizable robotic reward functions from” in-the-wild” human videos.  _arXiv preprint arXiv:2103.16817_ , 2021.
  * Dayan (1993) Peter Dayan.  Improving generalization for temporal difference learning: The successor representation.  _Neural computation_ , 5(4):613–624, 1993.
  * Ernst et al. (2005) Damien Ernst, Pierre Geurts, and Louis Wehenkel.  Tree-based batch mode reinforcement learning.  _Journal of Machine Learning Research_ , 6:503–556, 2005\.
  * Eysenbach et al. (2021) Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov.  Replacing rewards with examples: Example-based policy search via recursive classification, 2021.
  * Eysenbach et al. (2022) Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine.  Contrastive learning as goal-conditioned reinforcement learning.  _arXiv preprint arXiv:2206.07568_ , 2022.
  * Finn et al. (2016) Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International conference on machine learning_ , pages 49–58. PMLR, 2016.
  * Fu et al. (2018a) Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adversarial inverse reinforcement learning.  _International Conference on Learning Representations_ , 2018a.
  * Fu et al. (2018b) Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine.  Variational inverse control with events: A general framework for data-driven reward definition.  _arXiv preprint arXiv:1805.11686_ , 2018b.
  * Fujimoto and Gu (2021) Scott Fujimoto and Shixiang Shane Gu.  A minimalist approach to offline reinforcement learning.  _arXiv preprint arXiv:2106.06860_ , 2021.
  * Fujimoto et al. (2018) Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  _arXiv preprint arXiv:1812.02900_ , 2018.
  * Ghasemipour et al. (2021) Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu.  Emaq: Expected-max q-learning operator for simple yet effective offline and online rl.  In _International Conference on Machine Learning_ , pages 3682–3691. PMLR, 2021.
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  _The elements of statistical learning: data mining, inference, and prediction_ , volume 2.  Springer, 2009.
  * Ho and Ermon (2016) Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  _Conference on Neural Information Processing Systems_ , 2016.
  * Hoffman et al. (2020) Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al.  Acme: A research framework for distributed reinforcement learning.  _arXiv preprint arXiv:2006.00979_ , 2020.
  * Jaques et al. (2019) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv preprint arXiv:1907.00456_ , 2019.
  * Jarboui and Perchet (2021) Firas Jarboui and Vianney Perchet.  Offline inverse reinforcement learning, 2021.  URL https://arxiv.org/abs/2106.05068.
  * Julian et al. (2020) Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman.  Efficient adaptation for end-to-end vision-based robotic manipulation.  _arXiv preprint arXiv:2004.10190_ , 2020.
  * Kaelbling (1993) Leslie Pack Kaelbling.  Learning to achieve goals.  In _IJCAI_ , pages 1094–1099. Citeseer, 1993.
  * Kalashnikov et al. (2018) Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning_ , pages 651–673. PMLR, 2018.
  * Kalashnikov et al. (2021) Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman.  Mt-opt: Continuous multi-task robotic reinforcement learning at scale.  _arXiv preprint arXiv:2104.08212_ , 2021.
  * Kidambi et al. (2020) Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims.  Morel: Model-based offline reinforcement learning.  _arXiv preprint arXiv:2005.05951_ , 2020.
  * Konyushkova et al. (2020) Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas.  Semi-supervised reward learning for offline reinforcement learning.  _Offline Reinforcement Learning Workshop at Neural Information Processing Systems_ , 2020.
  * Kostrikov et al. (2021) Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum.  Offline reinforcement learning with fisher divergence critic regularization.  _arXiv preprint arXiv:2103.08050_ , 2021.
  * Kumar et al. (2019) Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  In _Advances in Neural Information Processing Systems_ , pages 11761–11771, 2019.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _arXiv preprint arXiv:2006.04779_ , 2020.
  * Lange et al. (2012) Sascha Lange, Thomas Gabel, and Martin A. Riedmiller.  Batch reinforcement learning.  In _Reinforcement Learning_ , volume 12. Springer, 2012.
  * Lee et al. (2021) Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim.  Representation balancing offline model-based reinforcement learning.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=QpNz8r_Ri2Y.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Li et al. (2021) Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine.  Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning.  In _International Conference on Machine Learning_ , pages 6346–6356. PMLR, 2021.
  * Liu et al. (2020) Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill.  Provably good batch reinforcement learning without great exploration.  _arXiv preprint arXiv:2007.08202_ , 2020.
  * Ma and Collins (2018) Zhuang Ma and Michael Collins.  Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency.  _arXiv preprint arXiv:1809.01812_ , 2018.
  * Mandlekar et al. (2020) Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox.  Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4414–4420. IEEE, 2020.
  * Matsushima et al. (2020) Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu.  Deployment-efficient reinforcement learning via model-based offline optimization.  _arXiv preprint arXiv:2006.03647_ , 2020.
  * Mazoure et al. (2020) Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm.  Deep reinforcement and infomax learning.  _Advances in Neural Information Processing Systems_ , 33:3686–3698, 2020.
  * Nair et al. (2022) Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Ng and Russell (2000) Andrew Y. Ng and Stuart J. Russell.  Algorithms for inverse reinforcement learning.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , ICML ’00, 2000.
  * Peng et al. (2019) Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Plappert et al. (2018) Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_ , 2018.
  * Pomerleau (1988) Dean A Pomerleau.  Alvinn: an autonomous land vehicle in a neural network.  In _Proceedings of the 1st International Conference on Neural Information Processing Systems_ , pages 305–313, 1988.
  * Rafailov et al. (2021) Rafael Rafailov, Tianhe Yu, A. Rajeswaran, and Chelsea Finn.  Offline reinforcement learning from images with latent space models.  _Learning for Decision Making and Control (L4DC)_ , 2021.
  * Ratliff et al. (2006) Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich.  Maximum margin planning.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, 2006.
  * Reddy et al. (2020) Siddharth Reddy, Anca D. Dragan, and Sergey Levine.  Sqil: Imitation learning via reinforcement learning with sparse rewards.  _International Conference on Learning Representations_ , 2020.
  * Riedmiller (2005) Martin Riedmiller.  Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method.  In _European Conference on Machine Learning_ , pages 317–328. Springer, 2005.
  * Ross et al. (2011) Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  _AISTATS_ , 2011.
  * Siegel et al. (2020) Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  _arXiv preprint arXiv:2002.08396_ , 2020.
  * Singh et al. (2019) Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine.  End-to-end robotic reinforcement learning without reward engineering.  _arXiv preprint arXiv:1904.07854_ , 2019.
  * Singh et al. (2020) Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine.  Cog: Connecting new skills to past experience with offline reinforcement learning.  _arXiv preprint arXiv:2010.14500_ , 2020.
  * Sinha and Garg (2021) Samarth Sinha and Animesh Garg.  S4rl: Surprisingly simple self-supervision for offline reinforcement learning.  _arXiv preprint arXiv:2103.06326_ , 2021.
  * Spencer et al. (2021) Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell.  Feedback in imitation learning: The three regimes of covariate shift.  _ArXiv Preprint_ , 2021.
  * Swazinna et al. (2020) Phillip Swazinna, Steffen Udluft, and Thomas Runkler.  Overcoming model bias for robust offline deep reinforcement learning.  _arXiv preprint arXiv:2008.05533_ , 2020.
  * Whitehead and Ballard (1991) Steven D Whitehead and Dana H Ballard.  Learning to perceive and act by trial and error.  _Machine Learning_ , 7(1):45–83, 1991.
  * Wu et al. (2019) Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Xie et al. (2018) Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn.  Few-shot goal inference for visuomotor learning and planning.  In _Conference on Robot Learning_ , pages 40–52. PMLR, 2018.
  * Xu and Denil (2019) Danfei Xu and Misha Denil.  Positive-unlabeled reward learning.  _arXiv preprint arXiv:1911.00459_ , 2019.
  * Xu et al. (2022) Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin.  Discriminator-weighted offline imitation learning from suboptimal demonstrations.  _International Conference on Machine Learning_ , 2022.
  * Yu et al. (2020a) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on Robot Learning_ , pages 1094–1100. PMLR, 2020a.
  * Yu et al. (2020b) Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.  Mopo: Model-based offline policy optimization.  _arXiv preprint arXiv:2005.13239_ , 2020b.
  * Yu et al. (2021) Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.  Combo: Conservative offline model-based policy optimization.  _arXiv preprint arXiv:2102.08363_ , 2021.
  * Yu et al. (2022) Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine.  How to leverage unlabeled data in offline reinforcement learning.  _International Conference on Machine Learning_ , 2022.
  * Zhou et al. (2020) Wenxuan Zhou, Sujay Bajracharya, and David Held.  Plas: Latent action space for offline reinforcement learning.  _arXiv preprint arXiv:2011.07213_ , 2020.
  * Zhu et al. (2020) Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine.  The ingredients of real-world robotic reinforcement learning.  _arXiv preprint arXiv:2004.12570_ , 2020.
  * Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  * Zolna et al. (2020a) Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed.  Offline learning from demonstrations and unlabeled experience.  _arXiv preprint arXiv:2011.13885_ , 2020a.
  * Zolna et al. (2020b) Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarejo, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang.  Task-relevant adversarial imitation learning.  _Conference on Robot Learning_ , 2020b.

##  Appendix A Proofs

The proof follows by substituting Assumption 1 into the definition of Q-values
(Eq. 1):

######  Proof A.1.

$\displaystyle
Q^{\beta}(s,a)=\mathbb{E}_{\beta(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t})\bigg{|}\begin{subarray}{c}s_{0}=s\\\
a_{0}=a\end{subarray}\right]$ $\displaystyle=\frac{1}{1-\gamma}\int
p^{\beta}(s_{t+}=s^{*}\mid s,a)r(s^{*})ds^{*}$
$\displaystyle=\frac{1}{1-\gamma}\int p^{\beta}(s_{t+}=s^{*}\mid
s,a)c\frac{p_{*}(s^{*})}{p_{\tau}(s^{*})}ds^{*}$
$\displaystyle=\frac{c}{1-\gamma}\int
p_{*}(s^{*})e^{f(s,a,s^{*})}ds^{*}=\frac{c}{1-\gamma}\mathbb{E}_{s^{*}\sim
p_{*}(s)}\left[e^{f(s,a,s^{*})}\right].$

##  Appendix B Experimental Details

We implemented our method and all baselines using the ACME framework (Hoffman
et al., 2020).

  * •

Batch size: 1024 for state based experiments, 256 for image based experiments

  * •

Training iterations: $250,000$ if task success rates had converged by that
point, otherwise $500,000$

  * •

Representation dimension: 256

  * •

Reward learning loss (for baselines): binary cross entropy (for ORIL) and
positive unlabeled (for PURL)

  * •

Critic architecture: Two-layer MLP with hidden sizes of 1024. ReLU activations
used between layers.

  * •

Reward function architecture (for baselines): Two-layer MLP with hidden sizes
of 1024. ReLU activations used between layers.

  * •

Actor learning rate: $3\times 10^{-4}$

  * •

Critic learning rate: $3\times 10^{-4}$

  * •

Reward learning rate (for baselines): $1\times 10^{-4}$

  * •

$\lambda$ for behavioral cloning weight in policy loss term: $0.5$

  * •

$\eta$ for PU loss: $0.5$

  * •

Size of offline datasets: Each dataset on Fetch tasks the consists of
approximately $4,000$ trajectories of length $50$, except for the FetchPush-
image dataset, which consists of approximately $40,000$ trajectories. Each
dataset on the Sawyer tasks consists of approximately $4,000$ trajectories of
length $200$.

##  Appendix C Multitask Critic Experiments

Figure 10: Multitask Drawer Environment.: We apply cross entropy method (CEM)
optimization on the LAEO dynamics model trained only on the data from the
drawer close task to solve six different tasks: close, half-closed, open,
reach (near), reach (medium), and reach (far). .

The Half-closed task requires the agent to push the drawer from an open
position into a halfway closed position. The Open task requires the agent to
pull the drawer from a closed position into an open position. The Close task
is the same as in the original SawyerDrawerClose environment, and requires the
agent to push the drawer from an opened position into a closed position. The
three reaching tasks, Reach-near, Reach-medium, Reach-far, require the agent
to reach the end-effector to a three different target positions. The tasks are
visualized in Figure 10.

For these experiments, we load the final checkpoint of a critic network from
the previous set of experiments (Comparison to Goal-Conditioned RL), and
select actions by using cross entropy method (CEM) optimization on the critic
network. By using CEM optimization, we do not need to train a separate policy
network for each of the tasks. Since the LAEO dynamics model is a multistep
dynamics model (meaning that the model predicts whether a goal state will be
reached sometime in the future and not just at the subsequent timestep) we are
able to use CEM directly with the dynamics model. Specifically, for each task,
we collect a success example using a scripted policy from Yu et al. (2020a).
Then, at each environment timestep $t$, we condition the LAEO dynamics model
on the success example, and then run CEM to choose an action that maximizes
the output of the dynamics network. At each timestep, we perform 10 iterations
of CEM, using a population size of $10,000$ and an elite population size of
$2,000$. Results are averaged across eight random seeds.


$a^{*}(t,x)=\frac{\mu(t,x)-r}{\gamma\sigma^{2}(t,x)}+\frac{\rho\nu(t,x)}{\gamma\sigma(t,x)}u^{(0)}_{x}(t,x)$
is the optimal policy for the classical case.

Recall that $V^{(\lambda)}$ is the value function of the original non-
exploratory problem under the deterministic policy $a^{(\lambda)}$. By the
Feynman–Kac formula, $V^{(\lambda)}$ satisfies the PDE:

$\displaystyle V^{(\lambda)}_{t}+$
$\displaystyle\Big{(}r+\big{(}\mu(t,x)-r\big{)}a^{\lambda}(t,x)\Big{)}wV^{(\lambda)}_{w}+\frac{1}{2}\sigma^{2}(t,x){a^{\lambda}}^{2}(t,x)w^{2}V^{(\lambda)}_{ww}$
(42)
$\displaystyle+m(t,x)V^{(\lambda)}_{x}+\frac{1}{2}\nu^{2}(t,x)V^{(\lambda)}_{xx}+\rho\nu(t,x)\sigma(t,x)a^{(\lambda)}(t,x)wV^{(\lambda)}_{wx}=\beta
V^{(\lambda)},$

with the terminal condition
$V^{(\lambda)}(T,w,x)=U(w)=\frac{w^{1-\gamma}-1}{1-\gamma}$. Conjecturing
$V^{(\lambda)}(t,w,x)=\frac{w^{1-\gamma}-1}{1-\gamma}\exp\\{\psi^{(\lambda)}(t,x)\\}+\varphi^{(\lambda)}(t,x)$
and putting it to (42), we obtain
$-\frac{1}{1-\gamma}\exp\\{\psi^{(\lambda)}(t,x)\\}+\varphi^{(\lambda)}(t,x)=-\frac{e^{-\beta(T-t)}}{1-\gamma}$,
where $\psi^{(\lambda)}$ satisfies

$\displaystyle\psi^{(\lambda)}_{t}$
$\displaystyle+m(t,x)\psi^{(\lambda)}_{x}+\frac{1}{2}\nu^{2}(t,x)(\psi^{(\lambda)}_{xx}+{\psi^{(\lambda)}_{x}}^{2})+(1-\gamma)\rho\nu(t,x)\sigma(t,x)a^{(\lambda)}(t,x)\psi^{(\lambda)}_{x}$
$\displaystyle+[(1-\gamma)r-\beta]+(1-\gamma)\big{[}(\mu(t,x)-r)a^{(\lambda)}(t,x)-\frac{\gamma}{2}\sigma^{2}(t,x){a^{(\lambda)}}^{2}(t,x)\big{]}=0,\
\psi^{(\lambda)}(T,x)=0.$

Recall that when $\lambda=0$, $\psi^{(0)}=u^{(0)}$, which motivates us to
expand
$\psi^{(\lambda)}=u^{(0)}+\lambda\phi^{(1)}+\lambda^{2}\phi^{(2)}+O(\lambda^{3})$.
Substituting this to the above equation, we deduce that $\phi^{(1)}$ satisfies

$\displaystyle\phi^{(1)}_{t}$
$\displaystyle+m(t,x)\phi^{(1)}_{x}+\frac{1}{2}\nu^{2}(t,x)(\phi^{(1)}_{xx}+2u^{(0)}_{x}\phi^{(1)}_{x})$
$\displaystyle+\frac{(1-\gamma)\rho\nu(t,x)}{\gamma\sigma(t,x)}\big{[}\mu(t,x)-r+\rho\nu(t,x)\sigma(t,x)u^{(0)}_{x}\big{]}\phi^{(1)}_{x}=0,\
\phi^{(1)}(T,x)=0.$

Because $\phi^{(1)}\equiv 0$ is a solution to this linear PDE, it can be
easily checked that (20) is the equation satisfied by $\phi^{(2)}$. The proof
is complete.

###  13.5 Proof of Theorem 5.1

We prove the inequality for $(t,w,x)=(0,w_{0},x_{0})$, the other cases being
completely the same. As before, denote $\mu_{t}=\mu(t,X_{t})$,
$\sigma_{t}=\sigma(t,X_{t})$ and
$\boldsymbol{\pi}_{t}=\boldsymbol{\pi}(t,X_{t})$ where $X$ is the factor
process with $X_{0}=x_{0}$. Recall that the recursive utility
$J^{\boldsymbol{\pi}}$ solves the BSDE (8). Introduce a new BSDE

$\left\\{\begin{aligned}
\mathrm{d}F^{\boldsymbol{\pi}}_{t}=&-(1-\gamma)\Bigg{\\{}\Big{[}\lambda\mathcal{H}(\boldsymbol{\pi}_{t})+r+(\mu_{t}-r)\operatorname{Mean}(\boldsymbol{\pi}_{t})\\\
&-\frac{\gamma}{2}\sigma_{t}^{2}\big{(}\operatorname{Mean}(\boldsymbol{\pi}_{t})^{2}+\operatorname{Var}(\boldsymbol{\pi}_{t})\big{)}\Big{]}F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}\operatorname{Mean}(\boldsymbol{\pi}_{t})Z_{t}^{F,\boldsymbol{\pi}}\Bigg{\\}}\mathrm{d}t+Z_{t}^{F,\boldsymbol{\pi}}\mathrm{d}B_{t}^{X},\\\
F^{\boldsymbol{\pi}}_{T}=&1.\end{aligned}\right.$ (43)

Applying Itô’s lemma to
$Y_{t}=\frac{{w_{t}^{\boldsymbol{\pi}}}^{1-\gamma}}{1-\gamma}F^{\boldsymbol{\pi}}_{t}-\frac{1}{1-\gamma}$,
we get

$\displaystyle\mathrm{d}Y_{t}=$
$\displaystyle-\lambda\mathcal{H}(\boldsymbol{\pi}_{t})[(1-\gamma)Y_{t}+1]\mathrm{d}t$
$\displaystyle+\frac{{w_{t}^{\boldsymbol{\pi}}}^{1-\gamma}}{1-\gamma}F^{\boldsymbol{\pi}}_{t}\begin{pmatrix}(1-\gamma)\sigma_{t}\operatorname{Mean}(\boldsymbol{\pi}_{t})+\rho
Z^{F,\boldsymbol{\pi}}_{t}\\\
(1-\gamma)\sigma_{t}\sqrt{\operatorname{Var}(\boldsymbol{\pi}_{t})}\\\
\sqrt{1-\rho^{2}}Z^{F,\boldsymbol{\pi}}_{t}\end{pmatrix}\cdot\mathrm{d}\mathbf{B}_{t},$

which is the same equation (8) including the terminal condition (with
$\lambda_{t}\equiv\lambda$) satisfied by $J^{\boldsymbol{\pi}}$ . The
existence and uniqueness of the solution of (8) implies that
$J^{\boldsymbol{\pi}}_{t}\equiv
Y_{t}\equiv\frac{{w_{t}^{\boldsymbol{\pi}}}^{1-\gamma}}{1-\gamma}F^{\boldsymbol{\pi}}_{t}-\frac{1}{1-\gamma}$.
Moreover, because (43) is a linear BSDE with a positive terminal value, it
follows from El Karoui et al. (1997, Proposition 2.2) that
$F_{t}^{\boldsymbol{\pi}}>0$ almost surely.

Define a portfolio process

$\hat{\boldsymbol{\pi}}_{t}=\mathcal{N}\Big{(}\frac{(\mu_{t}-r)F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}Z^{F,\boldsymbol{\pi}}_{t}}{\gamma\sigma^{2}_{t}F_{t}^{\boldsymbol{\pi}}},\frac{\lambda}{\gamma\sigma_{t}^{2}}\Big{)}.$
(44)

Under this control, the recursive utility is
$J^{\hat{\boldsymbol{\pi}}}_{t}=\frac{{W_{t}^{\hat{\boldsymbol{\pi}}}}^{1-\gamma}}{1-\gamma}F^{\hat{\boldsymbol{\pi}}}_{t}-\frac{1}{1-\gamma}$,
where $F_{t}^{\hat{\boldsymbol{\pi}}}$ satisfies a similar BSDE as (43) with
$\boldsymbol{\pi}_{t}$ replaced by $\hat{\boldsymbol{\pi}}_{t}$.

Because the BSDE (43) is Markovian whose driver depends on $(t,X_{t})$ only,
we conjecture that its solution has the representation
$F_{t}^{\boldsymbol{\pi}}=v^{\boldsymbol{\pi}}(t,X_{t})$ and
$Z_{t}^{F,\boldsymbol{\pi}}=v_{x}^{\boldsymbol{\pi}}(t,X_{t})\nu(t,X_{t})$,
where $v^{\boldsymbol{\pi}}$ is some smooth function of $(t,x)$. Itô’s formula
validates this conjecture for which $v^{\boldsymbol{\pi}}$ satisfies some PDE
similar to that in the Feynman–Kac formula. Now, since
$F_{t}^{\boldsymbol{\pi}}\geq 0$, we can write
$v^{\boldsymbol{\pi}}(t,X_{t})=e^{u^{\boldsymbol{\pi}}(t,X_{t})}$. Then (44)
can be written as $\hat{\boldsymbol{\pi}}_{t}=\hat{\boldsymbol{\pi}}(t,X_{t})$
where $\hat{\boldsymbol{\pi}}(\cdot,\cdot)$ is given by (28).

It remains to prove that the portfolio process $\hat{\boldsymbol{\pi}}_{t}$
improves $\boldsymbol{\pi}_{t}$. Note that given a mean and a variance, the
Gaussian distribution maximizes the entropy; so
$\mathcal{H}(\boldsymbol{\pi}_{t})=\frac{1}{2}\log(2\pi
e\operatorname{Var}(\boldsymbol{\pi}_{t}))$. Hence we have the following
inequality:

$\displaystyle\Bigg{\\{}\Big{[}\lambda\mathcal{H}(\hat{\boldsymbol{\pi}}_{t})+r+(\mu_{t}-r)\operatorname{Mean}(\hat{\boldsymbol{\pi}}_{t})-\frac{\gamma}{2}\sigma_{t}^{2}\big{(}\operatorname{Mean}(\hat{\boldsymbol{\pi}}_{t})^{2}+\operatorname{Var}(\hat{\boldsymbol{\pi}}_{t})\big{)}\Big{]}F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}\operatorname{Mean}(\hat{\boldsymbol{\pi}}_{t})Z_{t}^{F,\boldsymbol{\pi}}\Bigg{\\}}$
$\displaystyle-\Bigg{\\{}\Big{[}\lambda\mathcal{H}(\boldsymbol{\pi}_{t})+r+(\mu_{t}-r)\operatorname{Mean}(\boldsymbol{\pi}_{t})-\frac{\gamma}{2}\sigma_{t}^{2}\big{(}\operatorname{Mean}(\boldsymbol{\pi}_{t})^{2}+\operatorname{Var}(\boldsymbol{\pi}_{t})\big{)}\Big{]}F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}\operatorname{Mean}(\boldsymbol{\pi}_{t})F_{t}^{F,\boldsymbol{\pi}}\Bigg{\\}}$
$\displaystyle\geq$ $\displaystyle\Bigg{\\{}\Big{[}\frac{\lambda}{2}\log 2\pi
e\operatorname{Var}(\hat{\boldsymbol{\pi}_{t}})-\frac{\gamma}{2}\sigma_{t}^{2}\operatorname{Var}(\hat{\boldsymbol{\pi}}_{t})\Big{]}F_{t}^{\boldsymbol{\pi}}+\Big{[}(\mu_{t}-r)F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}Z_{t}^{F,\boldsymbol{\pi}}\Big{]}\operatorname{Mean}(\hat{\boldsymbol{\pi}}_{t})-\frac{\gamma}{2}\sigma_{t}^{2}\operatorname{Mean}(\hat{\boldsymbol{\pi}}_{t})^{2}F_{t}^{\boldsymbol{\pi}}\Bigg{\\}}$
$\displaystyle-\Bigg{\\{}\Big{[}\frac{\lambda}{2}\log 2\pi
e\operatorname{Var}(\boldsymbol{\pi}_{t})-\frac{\gamma}{2}\sigma_{t}^{2}\operatorname{Var}(\boldsymbol{\pi}_{t})\Big{]}F_{t}^{\boldsymbol{\pi}}+\Big{[}(\mu_{t}-r)F_{t}^{\boldsymbol{\pi}}+\rho\sigma_{t}Z_{t}^{F,\boldsymbol{\pi}}\Big{]}\operatorname{Mean}(\boldsymbol{\pi}_{t})-\frac{\gamma}{2}\sigma_{t}^{2}\operatorname{Mean}(\boldsymbol{\pi}_{t})^{2}F_{t}^{\boldsymbol{\pi}}\Bigg{\\}}$
$\displaystyle\geq$ $\displaystyle 0.$

Based on the comparison theorem for linear BSDEs (El Karoui et al., 1997,
Theorem 2.2), we conclude that $F_{t}^{\hat{\boldsymbol{\pi}}}\geq
F_{t}^{\boldsymbol{\pi}}$ when $1-\gamma>0$, and
$F_{t}^{\hat{\boldsymbol{\pi}}}\leq F_{t}^{\boldsymbol{\pi}}$ when
$1-\gamma<0$. Consequently, $J_{t}^{\hat{\boldsymbol{\pi}}}\geq
J_{t}^{\boldsymbol{\pi}}$ for any $\gamma>0,\gamma\neq 1$.

###  13.6 Proof of Theorem 9.1

Consider an ODE $y^{\prime}=F(y)$ with the initial condition $y(0)=1$, where
$F(x)=ax+b-c\log x$ with some constants $a,b$ and $c$. Let
$y(\tau)=e^{-z(\tau)}$. Then $z$ satisfies

$z^{\prime}=-e^{z}F(e^{-z})=-a-be^{z}-cze^{z},\ z(0)=0.$ (45)

We have the following result.

###### Lemma 13.1

If $F(x)<0$ for any $x\in(0,1]$, then the solution of (45) explodes at time
$\tau_{e}=-\int_{0}^{\infty}\frac{1}{a+be^{z}+cze^{z}}\mathrm{d}z$.

Proof: Note that the condition $F<0$ implies that $z$ is increasing. The
explosion time is then given by the Osgood test (c.f., Ceballos-Lira et al.
2010). $\hfill\Box$

The desired result follows from Lemma 13.1 and the fact that $y\to 0$ is
equivalent to $z\to\infty$.

###  13.7 Proof of Theorem 11.1

By Itô’s formula, the solution to BSDE (43) can be represented by
$F_{t}^{\boldsymbol{\pi}}=e^{Y_{t}^{\boldsymbol{\pi}}}$, where
$Y^{\boldsymbol{\pi}}_{t}$ solves the following BSDE

$\left\\{\begin{aligned}
\mathrm{d}Y^{\boldsymbol{\pi}}_{t}=&-\Bigg{\\{}(1-\gamma)\Big{[}\lambda\mathcal{H}(\boldsymbol{\pi}_{t})+r+(\mu_{t}-r+\rho\sigma_{t}Z^{\boldsymbol{\pi}}_{t})\operatorname{Mean}(\boldsymbol{\pi}_{t})\\\
&-\frac{\gamma}{2}\sigma_{t}^{2}\big{(}\operatorname{Mean}(\boldsymbol{\pi}_{t})^{2}+\operatorname{Var}(\boldsymbol{\pi}_{t})\big{)}\Big{]}+\frac{1}{2}{Z_{t}^{\boldsymbol{\pi}}}^{2}\Bigg{\\}}\mathrm{d}t+Z_{t}^{\boldsymbol{\pi}}\mathrm{d}B_{t}^{X}\\\
Y^{\boldsymbol{\pi}}_{T}=&0,\end{aligned}\right.$ (46)

with
$F_{t}^{\boldsymbol{\pi}}Z_{t}^{\boldsymbol{\pi}}=Z_{t}^{F,\boldsymbol{\pi}}$.
Then the improved control (44) can be rewritten as

$\hat{\boldsymbol{\pi}}_{t}=\mathcal{N}\left(\frac{\mu_{t}-r+\rho\sigma_{t}Z_{t}^{\boldsymbol{\pi}}}{\gamma\sigma_{t}^{2}},\frac{\lambda}{\gamma\sigma_{t}^{2}}\right).$

Now, if $\boldsymbol{\pi}_{t}=\boldsymbol{\pi}^{*}_{t}$ is the optimal
control, then the improved control
$\hat{\boldsymbol{\pi}}_{t}^{*}\equiv\boldsymbol{\pi}^{*}_{t}$, leading to
$\operatorname{Mean}(\boldsymbol{\pi}^{*}_{t})=\frac{\mu_{t}-r+\rho\sigma_{t}Z_{t}^{\boldsymbol{\pi}^{*}}}{\gamma\sigma_{t}^{2}}$,
$\operatorname{Var}(\boldsymbol{\pi}^{*}_{t})=\frac{\lambda}{\gamma\sigma_{t}^{2}}$,
and $\mathcal{H}(\boldsymbol{\pi}^{*}_{t})=\frac{1}{2}\log\frac{2\pi
e\lambda}{\gamma\sigma_{t}^{2}}$. Plugging these three conditions to (46) with
$\boldsymbol{\pi}_{t}=\boldsymbol{\pi}^{*}_{t}$ results in the same BSDE as
(37). This completes our proof.


# Quantum information entropy of a particle trapped by the Aharonov-Bohm-type
effect

F. C. E. Lima<EMAIL_ADDRESS>Universidade Federal do Ceará,
Departamento do Física, Fortaleza, CE, 60455-760, Brazil.  A. R. P. Moreira
Universidade Federal do Ceará, Departamento do Física, Fortaleza, CE,
60455-760, Brazil.  C. A. S. Almeida<EMAIL_ADDRESS>Universidade
Federal do Ceará, Departamento do Física, Fortaleza, CE, 60455-760, Brazil.
C. O. Edet<EMAIL_ADDRESS>Institute of Engineering Mathematics,
Universiti Malaysia Perlis, 02600 Arau, Perlis, Malaysia.  Faculty of
Electronic Engineering Technology, Universiti Malaysia Perlis, 02600 Arau,
Perlis, Malaysia.  Department of Physics, Cross River University of
Technology, Calabar, Nigeria.  N. Ali<EMAIL_ADDRESS>Faculty of
Electronic Engineering Technology, Universiti Malaysia Perlis, 02600 Arau,
Perlis, Malaysia.  Advanced Communication Engineering (ACE) Centre of
Excellence, Universiti Malaysia Perlis, 01000 Kangar, Perlis, Malaysia.

###### Abstract

Abstract: In this research article, we use the Shannon’s formalism to
investigate the quantum information entropy of a particle trapped by the
Aharonov-Bohm-type field. For quantum information study, it is necessary to
investigate the eigenstates of the quantum system, i. e., the wave functions
and energies of the quantum states. We assumed that the particle is in
principle, confined in a cylindrical box in the presence of Aharonov-Bohm-type
effect due to dislocation defect. Analysis of the quantum information entropy,
reveals that the dislocation influences the eigenstates and, consequently, the
quantum information of the system.

Keywords: Shannon’s entropy, Aharonov-Bohm-type effect, Topological defect.

##  I Introduction

Vector and scalar potentials appear in physics as auxiliary tools describing
the electric and magnetic fields AB , Arfken . Furthermore, potentials play a
fundamental role in the canonical description of a physical system at the
classical Marion  and quantum levels Griffiths , Sakurai . In quantum theory,
potentials are responsible for the emergence of the Aharonov-Bohm (AB) effect
AB . The AB effect is a quantum phenomenon in which an electrically charged
particle is affected by an electromagnetic potential Griffiths , Sakurai . In
principle, the AB effect tells us that the particle will feel the influence of
the field even when the particle is at a null-field region AB , Griffiths ,
Sakurai . This effect was described in 1959 by Yakir Aharonov and David Bohm
in their seminal paper Significance of Electromagnetic Potentials in Quantum
Theory AB .

Several applications discussing AB effects have appeared in the literature
Overstreet , Ronen , Mefford , FAhmed1 , FAhmed2 , FAhmed3 , Ba1 , Ba2 , Ba3 .
For example, the AB effect arises when studying carbon nanotubes Ajiki , in
the scattering process Ruij , and in photonic systems Fang . The fact is that
there is a growing interest in the study of the AB effect Vaidman , Caprez ,
Chaichian . In particular, this interest motivates us to understand the
quantum information of a particle trapped by an AB-type effect due to the
presence of a cosmic string topological defect, i. e., the dislocation defect.

Cosmic strings are topological defects analogous to the flux tubes that arise
in type II superconductors and vortex filaments in superfluid helium. The Ref.
Vilenkin  suggests that this defect must have formed in a large unification
transition or simply conceivable later in the electroweak transition. It is
necessary to mention that the standard model does not predict stable strings.
However, some generalizations allow this prediction Vilenkin . Generally, the
cosmic string is a hypothetical topological defect formed during a symmetry-
breaking phase transition in the early universe when the topology of the
vacuum manifold associated with this symmetry-breaking was not simply
connected. The existence of cosmic string defect was proposed first by Kibble
in the seminal paper topology of cosmic domains and strings Kibble , Kibble2 .
For more details, see Refs. Kibble , Kibble2 , Helliwell , Gott , Linet .

The idea of quantum information arises in search of a general communication
theory Shannon . Claude E. Shannon sought an extension of communication theory
to include new factors. In particular, Shannon sought to implement in his
study the noise effect on the channel and the possible savings in the
propagation process of the message Shannon . This relevant study made Shannon
one of the forerunners of communication theory.

Shannon’s theory has been used in quantum mechanics to study information from
physical systems Collins , LMA , LMMA , Almeida1 . In quantum theory,
Shannon’s theory is a theoretical measuring of the communication responsible
for informing us of the measure of uncertainty in the position and momentum of
the particle LMA , LMMA . In other words, Shannon’s information is information
theoretical-measuring associated with the position space and reciprocal.
Indeed, this informational analysis plays a relevant role in the foundations
of information theory Shannon  and quantum mechanics problems Pathria ,
Griffiths , Sakurai  (see Refs. Dong1 , Dong2 , Dong3 ). Furthermore,
Shannon’s entropy is a useful approach used in the study of cryptography
Grosshans , noise theory Wyner , harmonic oscillators Boumali , Yanez ,
Dehesa0 , potential wells Bouvrie , Torres , Dehesa , Sun1 , Majernik ,
Tserkis , Mukherjee , Kumar , Amadi , and theories of effective mass Serrano ,
Hua , Navarro , Lima .

Some studies are found in the literature discussing the influence of the AB
effect on the system eigenstates Ikot0 . Furthermore, some studies were also
performed on the information of quantum-mechanical systems with AB effect with
a specific interaction Collins . However, as far as we know, no studies have
been carried out purely studying the influence of the AB-type effect on the
quantum information of a spinless particle. Therefore, the main purpose is to
study how quantum information changes as the AB-type effect is modified.

We organize the paper as follows: In Sec. II, we build the quantum theory and
present the analytical solution of the eigenstates of the system. Posteriorly,
in Sec. III, the quantum information of the system is studied considering an
alteration of the AB-type effect. Finally, in Sec. IV, we announce our
findings.

##  II Quantum description of the system

Our purpose in this section is the quantum description of a spinless particle
subjected to the AB effect in the presence of dislocation. So, the eigenstates
depend on the AB effect and the dislocation defect. In summary, the
dislocation defect corresponds to the distortion of a vertical line into a
spiral Valanis:2005pc , Puntigam:1996vy , daSilva:2019lzh . Seeking to
understand how the AB effect and dislocation modify the eigenstates, let us
consider a quantum system composed of a spinless particle in the presence of a
dislocation, i. e.,

$\displaystyle ds^{2}=dr^{2}+2\beta\,dr\,d\theta+r^{2}d\theta^{2}+dz^{2},$ (1)

where the parameter $\beta$ describes the dislocation and obeys the constraint
$0<\beta<1$. Furthermore, the cylindrical symmetry considered (1) requires
that $0<r<\infty$, $0\leq\theta\leq 2\pi$ and $-\infty\leq z\leq\infty$.

One classifies the topological defects into three classes, i.e., domain walls,
cosmic strings, and monopoles, see Ref. Vachaspati0 . In this article, metric
(1) describes a cosmic string defect. In particular, cosmic strings are linear
spacetime topological defects Vachaspati0 , Helliwell , Gott , Linet , Kibble
, Kibble2 , Vilenkin . Physically, we characterize this class of defects by a
conical singularity determined by a concentrated curvature on the symmetry
axis. Due to the relevant physical repercussions of concepts related to this
defect class, several studies have considered cosmic string defects in their
investigations. For more details, see Refs. Ramberg , WAhmed , HChen , KBakkeH
, BoumaliNM , HassanabadiHM , ZWang .

Considering a nonrelativistic particle in the presence of dislocation, we
write the Schrödinger equation as

$\displaystyle-\frac{1}{2m}\frac{1}{\sqrt{g}}\partial_{i}(g^{ij}\sqrt{g}\partial_{j})\psi(r,\theta,z)=\mathcal{E}\psi(r,\theta,z).$
(2)

Here, we use the $\hbar=c=1$.

Now using the line element (1) and considering the Eq. (2), one obtains

$\displaystyle-\frac{1}{2m}\bigg{\\{}\frac{\partial^{2}}{\partial
r^{2}}+\frac{\partial}{\partial
z^{2}}+\frac{1}{(r^{2}-\beta^{2})}\bigg{[}r\frac{\partial}{\partial
r}+\bigg{(}\frac{\partial}{\partial\theta}-\beta\frac{\partial}{\partial
z}\bigg{)}^{2}\bigg{]}\bigg{\\}}\psi(r,\theta,z)=\mathcal{E}\psi(r,\theta,z).$
(3)

To solve Schrödinger’s equation (3), allow us assume

$\displaystyle\psi(r,\theta,z)=\text{e}^{i(l\theta+kz)}R(r).$ (4)

Here, $l=0,\pm 1,\pm 2,\pm 3,...$ and $k\in\mathbb{R}$. Besides, note that the
parameters $l$ and $k$ are the eigenvalues associated with the operators
$\hat{p}_{z}=-i\partial_{z}$ and $\hat{L}_{z}=-i\partial_{\theta}$,
respectively.

Assuming the transformation (4) where $\theta$ and $z$ are cyclic coordinates,
we write Schrödinger’s equation as

$\displaystyle
R^{\prime\prime}(r)+\frac{1}{(r^{2}-\beta^{2})}\bigg{[}rR^{\prime}(r)-(l-\beta
k)^{2}R(r)\bigg{]}+(2m\mathcal{E}-k^{2})R(r)=0.$ (5)

Here, prime notation represents the derivative concerning the radial
coordinate.

Seeking to describe the confinement of the spinless particle, let us assume
the coordinate change

$\displaystyle r=\frac{x}{\sqrt{2m\mathcal{E}-k^{2}}}.$ (6)

This coordinate change appears in Refs. daSilva:2019lzh , MulerK  to study a
particle confinement by a hard wall.

Considering the Eq. (6), we rewrite the Eq. (5) as

$\displaystyle\bigg{(}1-\frac{4\varsigma^{2}}{x^{2}}\bigg{)}\ddot{R}(x)+\frac{1}{x}\dot{R}(x)+\bigg{(}1-\frac{\lambda}{x^{2}}-\frac{2\varsigma^{2}}{x^{2}}\bigg{)}R(x)=0,$
(7)

where

$\displaystyle\lambda=(l-\beta k)^{2}+\varsigma^{2},$ (8)

and

$\displaystyle\varsigma^{2}=\frac{\beta^{2}(2m\mathcal{E}-k^{2})}{4}.$ (9)

Here, the point notation represents the derivative concerning the variable
$x$.

Let us highlight that the parameter that describes the dislocation (or
distortion of the metric) has a value restrained in the range $[0,1]$
Valanis:2005pc , Puntigam:1996vy . So, one can assume the parameter $\beta$ is
small. Therefore, the higher order terms of $\beta$ will be negligible. In
this way, we reformulate the Eq. (7) as follows:

$\displaystyle\ddot{R}(x)+\frac{1}{x}\dot{R}(x)+\bigg{[}1-\bigg{(}\frac{l-\beta
k}{x}\bigg{)}^{2}\bigg{]}R(x)=0.$ (10)

Here, allow us to remember that every equation written in the form

$\displaystyle
f^{\prime\prime}(x)+\frac{1}{x}f^{\prime}(x)+\bigg{(}k^{2}-\frac{\alpha^{2}}{x^{2}}\bigg{)}f(x)=0$
(11)

is known as Bessel’s equation Arfken , Abramowitz , Butkov . Thus, the general
solution of the Bessel equation will be

$\displaystyle f(x)=c_{1}\,J_{\alpha}(x)+c_{2}\,J_{-\alpha}(x),$ (12)

where

$\displaystyle
J_{\alpha}(x)=\sum_{k=0}^{\infty}(-1)^{k}\frac{1}{k!\,\Gamma(\alpha+k+1)}\bigg{(}\frac{x}{2}\bigg{)}^{\alpha+2k}.$
(13)

For more details on the Bessel equation, see Refs. Arfken , Abramowitz ,
Butkov .

Comparing Eq. (11) and its solution (12) with Eq. (10) and requiring the wave
function to be normalizable, one obtains that the solution of Eq. (10) is

$\displaystyle R(x)=A_{0}J_{|l-\beta k|}(x).$ (14)

Here $A_{0}$ is the normalization constant, and $J_{|l-\beta k|}(x)$ is
Bessel’s function of first kind Arfken , Abramowitz , Butkov .

Building the general wave function that describes spinless particles in the
presence of disclination, one obtains

$\displaystyle\psi(r,\theta,z)=A_{0}\text{e}^{i(l\theta+kz)}J_{|l-\beta
k|}\bigg{(}\frac{2\varsigma}{\beta^{2}}r\bigg{)}.$ (15)

As discussed by Silva and Bakke daSilva:2019lzh , let us analyze the
confinement of the spinless particle by a hard-wall confining potential in the
elastic medium. For this, allow us to assume that

$\displaystyle\lim_{x\to
x_{0}}R(x)=0\,\,\,\,\,\,\text{with}\,\,\,\,\,\,x_{0}=\beta\sqrt{2m\mathcal{E}-k^{2}}.$
(16)

Applying the constraint (16), one obtains that

$\displaystyle\mathcal{E}_{n,l,k}=\frac{1}{2m}\bigg{(}\frac{\Theta_{n,|l-\beta
k|}}{r_{0}}\bigg{)}^{2}+\frac{k^{2}}{2m}.$ (17)

Here $\Theta_{n,|l-\beta k|}$ is the $n$-th zero of the $|l-\beta k|$-th
Bessel’s function Griffiths . Furthermore, $n=0,1,2,\dots$, and the quantum
number associated to the angular momentum is $l=0,\pm 1,\pm 2,\dots$.

The results of the Eqs. (15) and (17) expose how eigenstates change with
disclination. So, we expect the existence of a dependency on the quantum
information with the disclination defect. In the next section, we present a
study of the variation of Shannon information as the disclination changes.

Note that the parameters $n$ and $|l-\beta k|$ play the role of quantum
numbers associated with the radial and angular variables, respectively.
Therefore, this result allows us to perceive the influence of the defect on
the energy eigenvalues. Moreover, one notes this influence by a change in the
quantum number of the angular momentum. It is relevant to mention that this
change arises even if there is no interaction between the particle and the
defect. Indeed, this effect is called the Aharonov-Bohm-type effect and
appears due to the presence of the topological defect.

##  III Information theoretical measure: Shannon’s entropy

The growing interest in studies of information-theoretical measurements of
quantum-mechanical systems is evident Grosshans , Wyner , Boumali , Yanez ,
Dehesa0 , Bouvrie , Torres , Dehesa , Sun1 , Majernik , Tserkis , Mukherjee ,
Kumar , Amadi , Serrano , Hua , Navarro , Lima , EI . Initially, the entropic
measure of information comes up with an alternative proposal to the well-known
Heisenberg uncertainty relation Navarro , BBM . Among different approaches
used to measure information entropies is the so-called Shannon’s entropy
Navarro . In quantum theories, Shannon’s entropy plays an essential role in
uncertainty measurement. In particular, this entropy gives us a new version of
Heisenberg’s uncertainty relationship between the position and momentum spaces
BBM . This relation is

$\displaystyle S_{\textbf{r}}+S_{\textbf{p}}\geq D(1+\text{ln}\pi).$ (18)

Here, $D$ is the spatial dimension (in our case $D=3$), $S_{\textbf{r}}$ is
the information entropy at the position space, and $S_{\textbf{p}}$ is the
information entropy at the momentum space.

Using the definition of Shannon’s entropy, let us write the quantum
information entropy as

$\displaystyle
S_{\textbf{r}}=-\int_{-\infty}^{\infty}\int_{0}^{2\pi}\int_{0}^{\infty}\,|\psi(r,\theta,z)|^{2}\,\text{ln}\,|\psi(r,\theta,z)|^{2}\,r\,drd\theta
dz,$ (19)

and

$\displaystyle
S_{\textbf{p}}=-\int_{-\infty}^{\infty}\int_{0}^{2\pi}\int_{0}^{\infty}\,|\psi(p_{r},p_{\theta},p_{z})|^{2}\,\text{ln}\,|\psi(p_{r},p_{\theta},p_{z})|^{2}\,p_{r}\,dp_{r}dp_{\theta}dp_{z},$
(20)

where

$\displaystyle\psi(p_{r},p_{\theta},p_{z})=$
$\displaystyle\int_{0}^{\infty}r\,dr\int_{0}^{2\pi}d\theta\int_{-\infty}^{\infty}\,dz\,\psi(r,\theta,z)\,\text{e}^{-i2\pi
p_{r}r\cos(p_{\theta}-\theta)}\,\text{e}^{-i2\pi p_{z}z}.$ (21)

i. e., the three-dimensional Fourier transform in cylindrical coordinates. For
more details, see Refs. Arfken , Abramowitz , Butkov .

Shannon’s entropy helps us to calculate the entropic information measures. In
this context, Shannon’s entropy has performed a relevant role in several
studies. For example, one can apply this approach in investigating Aharonov-
Bohm rings Collins , quantum systems with double-well potentials Dong2 ,
hyperbolic interaction Torres , position-dependent mass theories Lima , Hua ,
and the Aharonov-Bohm effect CI .

Motivated by these applications, we will use this approach to study quantum
information theoretical measurements of a spinless particle. That particle is
in the presence of a screw displacement, which produces the Aharonov-Bohm-type
effect. It is important to mention that the study of theoretical measurements
of information of this system is the first step to understanding the gain (or
loss) of quantum information of particles that interact with this class of
topological defect. So, to achieve our purpose, let us consider the wave
eigenfunctions (15) and substitute in Eq. (19). This substitution leads us to
Shannon’s entropy in the position space. Subsequently, we perform Fourier’s
transform (21) of the eigenfunctions (15) and calculate Shannon’s entropy in
the reciprocal space using Eq. (20). The numerical results shown in the table
1 are obtained numerically solving the integrals (19) and (20) for the first
eigenstates. In Fig. 1, the probability densities at the position space for
the ground state and the first two excited states used in Shannon’s entropy
calculation are displayed. On the other hand, in Fig. 2, probability densities
at the reciprocal space used in Shannon’s entropy calculation are exposed.

(a) (b)

(c)

Figure 1: Probability density at the position space. (a) The case: $n=0$ and
$l=0$. (b) The case: $n=1$ and $l=-1$. (c) The case: $n=2$ and $l=-2$.

Analyzing Fig. 1, i. e., the probability density for the first three energy
levels, one notes that the more energetic states have greater amplitudes and
oscillations. In addition, the results presented in Fig. 1 show that by
increasing the influence of the topological defect responsible for the
Aharonov-Bohm type effect, the amplitudes of the probability density increase.
That suggests that the quantum information of the system increases as the
Aharonov-Bohm-type effect increases (i.e., as the parameter $\beta$
increases).

(a) (b)

(c)

Figure 2: Probability density at the reciprocal space. (a) The case: $n=0$ and
$l=0$. (b) The case: $n=1$ and $l=-1$. (c) The case: $n=2$ and $l=-2$.

Looking at Fig. 2, one can see that the probability density of the particle at
the reciprocal space is a Dirac delta-like distribution. This probability
density profile suggests minimal uncertainty in the momentum of the spinless
particle. Furthermore, with the probability densities shown in Figs. 1 and 2,
we obtain the numerical results of the table 1. This result tells us that the
quantum information decreases as the disclination increases. This decreasing
in information suggests an increase in uncertainty measurements associated
with particle position measurements. Besides, the quantum information at the
reciprocal space increases when the disclination is larger. Thus, suggesting
less uncertainty in the momentum measurement of the particle. Furthermore, one
notes that these results remain valid for quantum states with higher energy.
It is worth mentioning that in all cases, we have

$\displaystyle S_{r}+S_{p}\geq 6.43419,$ (22)

and therefore, the Bialynicki-Birula and Mycielski BBM  relationship is
preserved.

$n$ | $l$ | $\beta$ | $S_{\textbf{r}}$ | $S_{\textbf{p}}$ | $S_{\textbf{r}}+S_{\textbf{p}}$ | $3(1+\ln\pi)$
---|---|---|---|---|---|---
|  | 0.2 | 9.74631 | 0.06678 | 9.81309 |
0 | 0 | 0.4 | 9.74262 | 0.07558 | 9.81821 | 6.43419
|  | 0.8 | 9.74040 | 0.12158 | 9.86199 |
|  | 0.2 | 9.74435 | 0.05526 | 9.79961 |
| -1 | 0.4 | 9.74424 | 0.10408 | 9.84832 | 6.43419
|  | 0.8 | 9.74387 | 0.29596 | 10.03984 |
|  | 0.2 | 9.74483 | 0.00770 | 9.75254 |
1 | 0 | 0.4 | 9.74439 | 0.10381 | 9.84821 | 6.43419
|  | 0.8 | 9.74353 | 0.29641 | 10.03991 |
|  | 0.2 | 9.74410 | 0.01640 | 9.76050 |
| 1 | 0.4 | 9.74377 | 0.13777 | 9.88154 | 6.43419
|  | 0.8 | 9.74312 | 0.29938 | 10.04251 |
|  | 0.2 | 9.74461 | 0.02251 | 9.76713 |
| -2 | 0.4 | 9.74329 | 0.22803 | 9.97133 | 6.43419
|  | 0.8 | 9.74277 | 0.35949 | 10.10231 |
|  | 0.2 | 9.74518 | 0.02720 | 9.77238 |
| -1 | 0.4 | 9.74361 | 0.25583 | 9.99944 | 6.43419
|  | 0.8 | 9.74301 | 0.48609 | 10.22910 |
|  | 0.2 | 9.74482 | 0.04111 | 9.78593 |
2 | 0 | 0.4 | 9.74370 | 0.25717 | 10.00091 | 6.43419
|  | 0.8 | 9.74317 | 0.52424 | 10.26742 |
|  | 0.2 | 9.74435 | 0.04188 | 9.78623 |
| 1 | 0.4 | 9.74335 | 0.35605 | 10.09942 | 6.43419
|  | 0.8 | 9.74291 | 0.85920 | 10.60214 |
|  | 0.2 | 9.74399 | 0.07462 | 9.81861 |
| 2 | 0.4 | 9.74311 | 0.44406 | 10.18722 | 6.43419
|  | 0.8 | 9.74272 | 0.91082 | 10.65351 |

Table 1: Numerical result of Shannon’s entropy.

Finally, analyzing the numerical results presented in table 1, one notes that
the quantum information entropy at the position space is always greater than
at the momentum space. That is because the probability regions at the position
space (Fig. 1) are larger than the probability regions at the momentum space
(Fig. 2). Thus, the uncertainty measure related to the particle position is
always greater. Furthermore, we notice that the quantity
$S_{\textbf{r}}+S_{\textbf{p}}$ decreases when the topological defect
contribution decreases. That is because, in the absence of the topological
defect, we recover the usual problem of a free particle in a space with
cylindrical symmetry. So, in this limit, the particle’s uncertainty measures
in both spaces are minimal once the cosmic string defects (dislocation) do not
contribute in this case. Meantime, the uncertainty measures on the position
and momentum increase as the states become more energetic, i.e., when $n$ and
$l$ increase. That is a consequence of the particles, in this case, being in
more energetic states when $n$ and $l$ increase. In other words, the particles
have a momentum increase, and their measurement uncertainties concerning their
velocity and position suffer alterations. These results are very interesting
and allow us to conclude that Heisenberg’s uncertainty principle remains
preserved in our theory. To finish, we observe that the particle energy one
modifies for different values of $l$ ($l=2$ and $l=-2$). Thus, the uncertainty
measures related to the position and momentum of the particle will be
distinct.

##  IV Final remarks

In this research, we have employed Shannon’s entropy to study the quantum
information entropy of a non-relativistic particle trapped by the Aharonov-
Bohm-type effect. The Schrodinger equation in cylindrical coordinate is solved
in the presence of topological defect and the resulting Aharonov-Bohm effect
to obtain the wave functions and eigenvalues of quantum system. These
eigensolutions are employed to evaluate the Shannon entropy numerically. From
our analysis of the quantum information, it is possible to seen that the
dislocation influences the eigenstates and, consequently, the quantum
information of the system. It is interesting to point out that the famous
Bialynicki-Birula and Mycielski (BBM) relationship is satisfied as well. Going
forward, the results of this study will find applications in quantum
information processing and related areas.

## Authors Declaration

### Conflicts of interest/Competing interest

All the authors declared that there is no conflict of interest in this
manuscript.

### Acknowledgements

F. C. E. Lima is grateful to Coordenação de Aperfeiçoamento de Pessoal de
Nível Superior (CAPES), no 88887.372425/2019-00. C. A. S. Almeida thanks to
Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), no
309553/2021-0. C. O. Edet acknowledges eJDS (ICTP). This research was
performed partially under LRGS Grant LRGS/1/2020/UM/01/5/2 (9012-00009) Fault
Tolerant Photonic Quantum States for Quantum Key Distribution Provided by the
Ministry of Higher Education Malaysia (MOHE).

## Data Availability

One can find the datasets that support the findings of this study through the
article.

## References

  * [1] Y. Aharonov and D. Bohm, Phys. Rev. 115 (1959) 485.
  * [2] G. B. Arfken and H. J. Weber, Mathematical methods for physicists, (6th ed., Elsevier Academic Press, New York, 2005).
  * [3] J. B. Marion, Classical dynamics of particles and systems, (5th Revised ed., Cengage Learning, USA, 2003).
  * [4] D. J. Griffiths, Introduction to quantum mechanics, (Prentice Hall, USA, New Jersey, 1994).
  * [5] J. J. Sakurai and J. Napolitano, Modern quantum mechanics, (Addison-Wesley Publishing Company, USA, 1994).
  * [6] C. Overstreet, P. Asenbaum, J. Curti, M. Kim and M. A. Kasevich, Science 375 (2022) 226.
  * [7] Y. Ronen, T. Werkmeister, D. H. Najafabadi, A. T. Pierce, L. E. Anderson, Y. J. Shin, S. Y. Lee, Y. H. Lee, B. Johnson, K. Watanabe, T. Taniguchi, A. Yacoby and P. Kim, Nature 16 (2021) 563.
  * [8] E. Mefford and K. Suzuki, JHEP 05 (2021) 026.
  * [9] F. Ahmed, Europhys. Lett. 130 (2020) 40003.
  * [10] F. Ahmed, Advances in High Energy Physics 2020 (2020) 1-10.
  * [11] F. Ahmed, Europhys. Lett. 131 (2020) 30002.
  * [12] A. V. D. M. Maia and K. Bakke, Quant. Stud.: Mathematics and Foundations 10 (2023) 79.
  * [13] K. Bakke, Int. J. Mod. Phys. A 36 (2021) 2150136.
  * [14] S. L. R. Vieira and K. Bakke, Phys. Rev. A 101 (2020) 032102.
  * [15] H. Ajiki and T. Ando, Physica B 201 (1994) 349.
  * [16] S. N. M. Ruijsenaars, Ann. Phys. 146 (1983) 1-34.
  * [17] K. Fang, Z. Yu and S. Fan, Phys. Rev. Lett. 108 (2012) 153901.
  * [18] L. Vaidman, Phys. Rev. A 86 (2012) 040101(R).
  * [19] A. Caprez, B. Barwick and H. Batelaan, Phys. Rev. Lett. 99 (2007) 210401.
  * [20] M. Chaichian, P. Prešnajdera, M. M. Sheikh-Jabbaric and A. Tureanu, Phys. Lett. B 527 (2002) 149.
  * [21] A. Vilenkin, Phys. Rev. D 23 (1981) 852.
  * [22] T. W. B. Kibble, J. Phys. A 9 (1976) 1387.
  * [23] T. W. B. Kibble, Phys. Rep. 67 (1980) 183.
  * [24] T. M. Helliwell and D. A. Konkowski, Am. J. Phys. 55 (1987) 401.
  * [25] J. R. Gott, Ap. J. 288 (1985) 422.
  * [26] B. Linet, Gen. Rel. Grav. 17 (1985) 1109.
  * [27] C. E. Shannon, Bell Syst. Tech. J. 27 (1948) 623.
  * [28] C. O. Edet, F. C. E. Lima, C. A. S. Almeida, N. Ali and M. Asjad, Entropy, 24 (2022) 1059.
  * [29] F. C. E. Lima, A. R. P. Moreira and C. A. S. Almeida, Int. J. Quant. Chem. 121 (2021) e26645.
  * [30] F. C. E. Lima, A. R. P. Moreira, L. E. S. Machado and C. A. S. Ameida, Int. J. Quant. Chem. 121 (2021) e26749.
  * [31] C. A. S. Almeida, C. O. Edet, F. C. E. Lima, N. Ali and M. Asjad, Results in Physics 47 (2023) 106343.
  * [32] R. K. Pathria, Statistical Mechanics, (2nd ed., Butterworth Heinemann, Jordan Hill, Oxford, 1996).
  * [33] C. A. Gil-Barrera, R. S. Carrillo, G.-H. Sun and S. -H. Dong, Entropy 24 (2022) 604.
  * [34] G. -H. Sun, S. -H. Dong, K. D. Launey,T. Dytrych and J. P. Draayer, Int. J. Quant. Chem. 115 (2015) 891.
  * [35] G. -H. Sun,S. -H. Dong and N. Saad, Int. J. Quant. Chem. 525 (2013) 934.
  * [36] F. Grosshans, N. J. Cerf, Phys. Rev. Lett. 92 (2004) 047905.
  * [37] A. D. Wyner, S. Shamai, Proc. IEEE 86 (1998) 442.
  * [38] A. Boumali, M. Labidi, Mod. Phys. Lett. A 33 (2018) 1850033.
  * [39] R. J. Yañez, W. V. Assche and J. S. Dehesa, Phys. Rev. A 50 (1994) 3065.
  * [40] J. S. Dehesa, E. D. Belega, I. V. Toranzo and A. I. Aptekarev, Int. J. Quantum Chem. 119 (2019) e25977.
  * [41] P. A. Bouvrie, J. C. Angulo, J. S. Dehesa, Phys. A 390 (2011) 2215.
  * [42] R. Valencia-Torres, G. H. Sun and S. H. Dong, Phys. Scr. 90 (2015) 035205.
  * [43] J. S. Dehesa, A. Martínez-Finkelshtein, V. N. Sorokin, Mol. Phys. 104 (2006) 613.
  * [44] G. -H. Sun, M. A. Aoki, S. -H. Dong, Chin. Phys. B 22 (2013) 050302.
  * [45] V. Majernik, R. Charvot, E. Majernikova, J. Phys. A: Math. Gen. 32 (1999) 2207.
  * [46] S. T. Tserkis, C. C. Moustakidis, S. E. Massen, C. P. Panos, Phys. Lett. A 378 (2014) 497.
  * [47] N. Mukherjee, A. K. Roy, Int. J. Quantum Chem. 11, 2018, e25596.
  * [48] P. R. Kumar, G. Kumar, R. Kumar, A. Kumar, Int. J. Quantum Chem. 116 (2016) 1413.
  * [49] P. O. Amadi, A. N. Ikot, A. T. Ngiangia, U. S. Okorie, G. J. Rampho, H. Y. Abdullah, Int. J. Quantum Chem. 120 (2020) e26246.
  * [50] F. A. Serrano, B. J. Falaye, S. H. Dong, Phys. A Stat. Mech. Appl. 446 (2016) 152.
  * [51] S. Guo-Hua, D. Popov, O. Camacho-Nieto, S. H. Dong, Chin. Phys. B 24 (2015) 100303.
  * [52] G. Yañez-Navarro, G. H. Sun, T. Dytrych, K. D. Launey, S. H. Dong, J. P. Draayer, Ann. Phys. 348 (2014) 153.
  * [53] F. C. E. Lima, Ann. Phys. 442 (2022) 168906.
  * [54] C. O. Edet and A. N. Ikot, Eur. Phys. J. Plus 136 (2021) 432.
  * [55] A. N. Ikot, C. O. Edet, P. O. Amadi, U. S. Okorie, G. J. Rampho and H. Y. Abdullah, Eur. Phys. J. D 74 (2020) 159.
  * [56] K. S. Valanis and V. P. Panoskaltsis, Acta Mech. 175 (2005) 77.
  * [57] R. A. Puntigam and H. H. Soleng, Class. Quant. Grav. 14 (1997), 1129.
  * [58] W. C. F. da Silva and K. Bakke, Eur. Phys. J. C 79 (2019) 559.
  * [59] T. Vachaspati, Phys. Rev. D 44 (1991) 3723.
  * [60] N. Ramberg, W. Ratzinger and P. Schwaller, JCAP 02 (2023) 039.
  * [61] M. Yamada and K. Yonekura, Phys. Rev. D 106 (2022) 123515.
  * [62] W. Ahmed, M. Junaid, S. Nasri and U. Zubair, Phys. Rev. D 105 (2022) 115008.
  * [63] H. Chen, Z. W. Long, C. Y. Long, S. Zare and H. Hassanabadi, Int. J. Geom. Meth. Mod. Phys. 19 (2022) 2250133.
  * [64] K. Bakke and H. Mota, Gen. Rel. Grav. 52 (2020) 97.
  * [65] A. Boumali and N. Messai, Can. J. Phys. 95 (2017) 999.
  * [66] H. Hassanabadi, M. Hosseinpour and M. de Montigny, Eur. Phys. J. Plus 132 (2017) 541.
  * [67] Z. Wang, Z. W. Long, C. Y. Long and B. Q. Wang, Can. J. Phys. 95 (2017) 331.
  * [68] H. J. W. Müler-Kirsten, Introduction to quantum mechanics: Schrödinger equation and path integral (Word Scientific, Singapore, 2006).
  * [69] M. Abramowitz, I.A. Stegum, Handbook of mathematical functions (Dover Publications Inc., New York, 1965).
  * [70] E. Butkov, Mathematical Physics, (1st ed., Addison-Wesley, USA, 1973).
  * [71] A. L. Silva Netto, C. Chesman and C. Furtado, Phys. Lett. A 372 (2008) 3894.
  * [72] I. Bialynicki-Birula and J. Mycielski, Comm. Math. Phys. 44, (1975) 129.
  * [73] C. O Edet and A. N. Ikot, Eur. Phys. J. C 136 (2021) 432.
  * [74] V. B. Bezerra, J. Math. Phys. 38 (1997), 2553.
  * [75] A. I. Aptekarev, J. S. Dehesa, R. J. Yáñez, J. Math. Phys. 35 (1994) 4423.
  * [76] W. Van Assche, R. J. Yáñez, J. S. Dehesa, J. Math. Phys. 36 (1995) 4106.
  * [77] I. I. Hirschmann Jr., Amer. J. Math. 79 (1957) 152.