# Learning Representations for Control with Hierarchical Forward Models

Trevor McInroe
University of Edinburgh
<EMAIL_ADDRESS>
&Lukas Schäfer
University of Edinburgh
<EMAIL_ADDRESS>
&Stefano V. Albrecht
University of Edinburgh
<EMAIL_ADDRESS>

###### Abstract

Learning control from pixels is difficult for reinforcement learning (RL)
agents because representation learning and policy learning are intertwined.
Previous approaches remedy this issue with auxiliary representation learning
tasks, but they either do not consider the temporal aspect of the problem or
only consider single-step transitions. Instead, we propose Hierarchical
$k$-Step Latent (HKSL), an auxiliary task that learns representations via a
hierarchy of forward models that operate at varying magnitudes of step
skipping while also learning to communicate between levels in the hierarchy.
We evaluate HKSL in a suite of 30 robotic control tasks and find that HKSL
either reaches higher episodic returns or converges to maximum performance
more quickly than several current baselines. Also, we find that levels in
HKSL’s hierarchy can learn to specialize in long- or short-term consequences
of agent actions, thereby providing the downstream control policy with more
informative representations. Finally, we determine that communication channels
between hierarchy levels organize information based on both sides of the
communication process, which improves sample efficiency.

> Keywords: Reinforcement learning, Representation learning, Continuous
> control

##  1 Introduction

Recently, reinforcement learning (RL) has had significant empirical success in
the robotics domain [1, 2, 3, 4]. However, previous methods often require a
dataset of hundreds of thousands or millions of agent-environment interactions
to achieve their performance. This level of data collection may not be
feasible for the average industry group. Therefore, RL’s widespread real-world
adoption requires agents to learn a satisfactory control policy in the
smallest number of agent-environment interactions possible.

Pixel-based state spaces increase the sample efficiency challenge because the
RL algorithm is required to learn a useful representation and a control policy
simultaneously. A recent thread of research has focused on developing
auxiliary learning tasks to address this dual-objective learning problem.
These approaches aim to learn a compressed representation of the high-
dimensional state space upon which agents learn control. Several task types
have been proposed such as image reconstruction [5, 6], contrastive objectives
[7, 8], image augmentation [9, 10], and forward models [11, 12, 13, 14, 15].

Forward models are a natural fit for RL because they exploit the temporal axis
by generating representations of the state space that capture information
relevant to the environment’s transition dynamics. However, previous
approaches learn representations by predicting single-step transitions, which
may not capture relevant information efficiently if important environmental
changes take many steps to manifest. For example, if we wish to train a
soccer-playing agent to score a goal, the pertinent portions of an episode
occur at the beginning, when the agent applies a force and direction, and at
the end when the agent sees how close the ball came to the goal. Skipping the
large portion of the trajectory where the ball is rolling could lead to more
efficient learning.

In this paper, we introduce Hierarchical $k$-Step Latent
(HKSL)222https://github.com/trevormcinroe/hksl, an auxiliary task for RL
agents that explicitly captures information in the environment at varying
levels of temporal coarseness. HKSL accomplishes this by leveraging a
hierarchical latent forward model where each level in the hierarchy predicts
transitions with a varying number of steps skipped. Levels that skip more
steps should capture a coarser understanding of the environment by focusing on
changes that take more steps to manifest and vice versa for levels that skip
fewer steps. HKSL also learns to share information between levels via a
communication module that passes information from higher to lower levels. As a
result, HKSL learns a set of representations that give the downstream RL
algorithm information on both short- and long-term changes in the environment.

We evaluate HKSL and various baselines in a suite of 30 continuous-control
environments [16, 17] that contains environments without and with distractors
of varying types and intensities. We test our algorithms with and without
distractors because real-world RL-controlled robots need to work well in
controlled settings like a laboratory and uncontrolled settings like a public
street. Also, distractors may change at speeds independently from task-
relevant information, thereby increasing the challenge of relating agent
actions to changes in pixels. Therefore, real-world RL deployments should
explicitly learn representations that tie agent actions to long- and short-
term changes in the environment. The goal in our study is to learn a well-
performing control policy in the smallest number of agent-environment
interactions as possible.

Our experiments show that HKSL significantly outperforms current baselines in
terms of episodic returns in our evaluation suite. We analyze HKSL’s
hierarchical model and find that representations can learn to specialize in
varying levels of temporal coarseness, thereby providing the downstream
control policy with more informative representations. Additionally, we find
that HKSL’s communication manager considers both sides of the communication
process, thereby giving forward models information that better contextualizes
their learning process. Finally, we provide data from all training runs for
all benchmarked methods.

##  2 Background

We study an RL formulation wherein an agent learns a control policy within a
partially observable Markov decision process (POMDP) [18, 19], defined by the
tuple $(\mathcal{S},\mathcal{O},\mathcal{A},P^{s},P^{o},\mathcal{R},\gamma)$.
$\mathcal{S}$ is the ground-truth state space, $\mathcal{O}$ is a pixel-based
observation space, $\mathcal{A}$ is the action space,
$P^{s}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is the
state transition probability function,
$P^{o}:\mathcal{S}\times\mathcal{A}\times\mathcal{O}\rightarrow[0,1]$ is the
observation probability function,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward
function that maps states and actions to a scalar signal, and $\gamma\in[0,1)$
is a discount factor. The agent does not directly observe the state
$s_{t}\in\mathcal{S}$ at step $t$, but instead receives an observation
$o_{t}\in\mathcal{O}$ which we specify as a stack of the last three images. At
each step $t$, the agent samples an action $a_{t}\in\mathcal{A}$ with
probability given by its control policy which is conditioned on the
observation at time $t$, $\pi(a_{t}|o_{t})$. Given the action, the agent
receives a reward $r_{t}=\mathcal{R}(s_{t},a_{t})$, the POMDP transitions into
a next state $s_{t+1}\in\mathcal{S}$ with probability
$P^{s}(s_{t},a_{t},s_{t+1})$, and the next observation (stack of pixels)
$o_{t+1}\in\mathcal{O}$ is sampled with probability
$P^{o}(s_{t+1},a_{t},o_{t+1})$. Within this POMDP, the agent must learn a
control policy that maximizes the sum of discounted returns over the time
horizon $T$ of the POMDP’s episode:
$\operatorname*{arg\,max}_{\pi}\mathbb{E}_{a\sim\pi}[\sum_{t=1}^{T}\gamma^{t}r_{t}]$.

##  3 Related Work

Representation learning in RL. Some research has pinpointed the development of
representation learning methods that can aid policy learning for RL agents. In
model-free RL, using representation learning objectives as auxiliary tasks has
been explored in ways such as contrastive objectives [7, 8], image
augmentation [9, 10], image reconstruction [5], information theoretic
objectives [20], and inverse models [21, 22]. HKSL fits within the auxiliary
task literature but does not use contrastive objectives, image reconstruction,
information theoretic objectives, nor inverse models.

Forward models and hierarchical models. Forward models for model-free RL
approaches learn representations that capture the environment’s transition
dynamics via a next-step prediction objective. Some methods learn stochastic
models that are aided with image reconstruction [11] or reward-prediction
objectives [13]. Other methods combine deterministic forward models with
reward prediction and bisimulation metrics [12] or momentum regression targets
[23]. Outside of the purpose of representation learning, forward models are
used extensively in model-based RL approaches to learn control policies via
planning procedures [14, 15, 24, 25].

Stacking several forward models on top of one another forms the levels of a
hierarchical model. This type of model has been studied in the context of
multiscale temporal inference [26], variational inference [27], and pixel-
prediction objectives [28, 29]. Additionally, hierarchical models have been
used for speech synthesis [30], learning graph embeddings [31], and
decomposing MDPs [32]. Sequence prediction literature has explored the use of
hierarchical models via manually-defined connections between levels [29, 33]
and using levels with uniform time-step skipping [34, 35].

Unlike the aforementioned forward model approaches, HKSL combines a set of
forward models that step in the latent space with independent step sizes
without additional prediction objectives. Also, HKSL contains a connection
between forward models that learns what to share by using the context from the
entire rollout from higher levels and the current timestep of lower levels,
which leads to faster learning.

##  4 Hierarchical $k$-Step Latent

HKSL’s hierarchical model is comprised of forward models that take steps in
the latent space at varying levels of temporal coarseness. We define temporal
coarseness as the degree to which a level’s forward model skips environment
steps. For example, if a forward model predicts the latent representation of a
state five steps into the future, it is considered more coarse than a forward
model that predicts only one step forward. Coarser levels should learn to
attend to information in the environment that takes many steps to manifest in
response to an agent’s action. In contrast, finer levels should learn to
attend to environmental properties that immediately respond to agent actions.
This is because coarser levels need to make fewer predictions to reach steps
further into the future than finer levels.

At each learning step, a batch of $B$ trajectories of length $k$ are sampled
from the replay memory
$\tau=\\{(o_{t},a_{t},\dots,a_{t+k-1},o_{t+k})_{i}\\}_{i=1}^{B}$. The initial
observation of each trajectory $o_{t}$ is uniformly randomly sampled on a per-
episode basis $t\sim U(1,T-k)$222Ending the range of numbers on $T-k$
guarantees that trajectories do not overlap episodes.. In the following, we
will denote the first and last timestep of the batch with $t=1$ and $t=k$,
respectively.

HKSL’s components. See Figure 1 for a visual depiction of the HKSL
architecture. HKSL’s hierarchical model is comprised of $h$ levels. Each level
$l$ has a forward model $f^{l}$, a nonlinear projection module $w^{l}$, an
online image encoder $e_{o}^{l}$, and a momentum image encoder $e_{m}^{l}$
that is updated as an exponential moving average of the online encoder (e.g.,
[36]). Between consecutive levels there is a communication manager $c^{l,l-1}$
to pass information from one level $l$ to the level below it $l-1$. The number
of steps skipped by a given level $n^{l}$ is independent of the coarseness of
other levels in the hierarchy. Giving each level its own set of learnable
models allows specialization to occur at each magnitude of temporal
coarseness.

Figure 1: Depiction of HKSL architecture with an “unrolled” two-level
hierarchical model where the first level moves at one step $n^{1}=1$ and the
second level moves at three steps $n^{2}=3$. First, the online encoders (blue)
encode the initial observation $o_{1}$ of the sampled trajectory. Next, the
forward models (red) predict the latent representations of the following
observations, with level 1 predicting single steps ahead conditioned on the
level’s previous representation and applied action. The forward model of the
second level predicts three steps ahead and receives the previous
representation and concatenation of the three applied actions. The
communication manager (green) forwards information from the representations of
the coarser second level to each forward model step of the first level as
additional inputs. All models are trained end-to-end with a normalized
$\ell_{2}$ loss of the difference between the projected representations of
each level and timestep and the target representations of observations at the
predicted timesteps. Target representations are obtained using momentum
encoders (purple) and projections are done by the projection model (orange) of
the given level.

Forward models. HKSL’s forward models are a modified version of the common GRU
recurrent cell [37] that allows for multiple data inputs at each step. See
Section C.3 for a detailed mathematical description. At step $t=1$, the
forward models take the representation produced by the level’s encoder
$z_{1}^{l}=e_{o}^{l}(o_{1})$ along with a concatenation of $n^{l}$ action
vectors $\bar{a}_{1}=[a_{1}|...|a_{n^{l}}]$ to predict the latent
representation of a future state
$z^{l}_{1+n^{l}}=f^{l}(z^{l}_{1},\bar{a}_{1})$. For any following timestep
$t>1$, the forward models take the predicted latent representation $z^{l}_{t}$
as input instead of the encoder representation.

Communication managers. Communication managers $c^{l,l-1}$ pass information
from coarser to finer levels in the hierarchy ($l\rightarrow l-1$) while also
allowing gradients to flow from finer to coarser levels ($l-1\rightarrow l$).
A communication manager $c^{l,l-1}$ takes all latent representations produced
by level $l$ and one-hot-encoded step $t$ as inputs and extracts information
that is relevant for the forward model in level $l-1$ at step $t$. For all
levels other than the highest level in the hierarchy, the forward models also
receive the output of $c$.

Loss function. HKSL computes a loss value at each timestep within each level
in the hierarchy as the normalized $\ell_{2}$ distance between a nonlinear
projection of the forward model’s prediction and the “true” latent
representation produced by the level’s momentum encoder. Using this “noisy”
approximation of the target ensures smooth changes in the target between
learning steps and is hypothesized to reduce the possibility of collapsed
representations [38, 39]. We denote the projection model of level $l$ with
$w^{l}$ and the HKSL loss of level $l$ across the minibatch of trajectories
$\tau$ can be written as

$\mathcal{L}_{HKSL}^{l}=\sum^{N}_{t=1}\mathbb{E}_{a,o\sim\tau}\left\lVert
w^{l}\left(f^{l}(z_{t}^{l},\bar{a}_{t},c^{l+1,l}(\cdot))\right)-e^{l}_{m}(o_{t+n^{l}})\right\rVert_{2}^{2},$
(1)

where $N$ is the number of steps that a given level can take in $\tau$.

HKSL and SAC. We make a few adjustments to the base SAC algorithm to help HKSL
fit naturally. For one, we replace the usual critic with an ensemble of $h$
critics. Each critic and target critic in the ensemble receive the latent
representations produced by a given level’s encoder and momentum encoder,
respectively. We allow critics’ gradients to update their encoders’ weights,
and each critic is updated using $n$-step returns where $n$ corresponds to the
$n$ of the level within which the critic’s given encoder resides. By matching
encoders and critics in this way, we ensure encoder weights are updated by
gradients produced by targets of the same temporal coarseness.

Second, the actor receives a concatenation of the representations produced by
all online encoders. HKSL’s actors will make better-informed action selections
because they can consider information in the environment that moves at varying
levels of temporal coarseness. Finally, we modify the actor’s loss function to
use a sum of Q-values from all critics:

$\mathcal{L}_{actor}=-\mathbb{E}_{a\sim\pi,o\sim\tau}\left[\sum_{l=1}^{h}Q^{l}(o,a)-\alpha\log\pi(a|[e_{o}^{1}(o)|...|e_{o}^{h}(o)])\right].$
(2)

##  5 Experiments

We evaluate HKSL with a series of questions and compare it against several
relevant baselines. First, is HKSL more sample efficient in terms of agent-
environment interactions than other representation learning methods (§ 5.2)?
Second, what is the efficacy of each of HKSL’s components (§ 5.3)? Third, can
the levels in HSKL learn to specialize in different magnitudes of temporal
coarseness (§ 5.4)? Finally, what does does $c^{l,l-1}$ consider when
providing information to $l-1$ from $l$ (§ 5.4)?

###  5.1 Experimental Setup

Baselines. We use DrQ [9], CURL [7], and PI-SAC [20] as our baselines. DrQ
regularizes Q-value learning by averaging temporal difference targets across
several augmentations of the same images. CURL uses a contrastive loss similar
to CPC [40] to learn image embeddings. PI-SAC uses a Conditional Entropy
Bottleneck [41] auxiliary loss with both a forward and backward model to learn
a representation of observations that capture the environment’s transition
dynamics. All methods use SAC [42, 43] as the base RL algorithm and we use the
same encoder, critic, and actor architectures for each agent type to ensure a
fair comparison. Additionally, each method uses the same image augmentation.
See Appendix C for SAC and HKSL hyperparameters.

Environments. We use six continuous-control environments provided by MuJoCo
[44] via the DMControl suite [16, 45], a popular set of environments for
testing robotic control algorithms. Each of the six environments uses episodes
of length 1k environment steps and a set number of action repeats that
controls the number of times the environment is stepped forward with a given
action. See Appendix B for further information on the environments.

We use five variations of each environment for a total of 30 tasks. Four of
the variations use distractors provided by the Distracting Control Suite API
[17], and the fifth variation uses no distractors. We use the “color” and
“camera” distractors on both the “easy” and “medium” difficulty settings. The
color distractor changes the color of the agent’s pixels on each environment
step, and the camera distractor moves the camera in 3D space each environment
step. The difficulty setting controls the range of color values and the
magnitude of camera movement in each task222Refer to [17] for details..

###  5.2 Sample Efficiency

Training and evaluation procedure. In our training scheme, agents are allowed
to perform an RL and representation learning gradient update once per action
selection. Every 10k environment steps, we perform an evaluation checkpoint,
wherein the agent’s policy is sampled deterministically as the mean of the
produced action distribution and we collect the mean performance across 10
episodes. All methods are trained with a batch size of 128 for 100k
environment steps across five seeds.

Results. We use the “rliable” package [46] to plot statistically robust
summary metrics across all 30 tasks in our evaluation suite. Specifically,
Figure 2 shows the interquartile mean (IQM) (left) and the optimality gap
(middle) along with their 95% confidence intervals (CIs) that are generated
via stratified bootstrap sampling333For all plots, we performed at least 5,000
samples. at the 100k steps mark. Optimality gap measures the amount by which a
given algorithm fails to achieve a perfect score222We note that a perfect
score (optimality gap = 0) is technically impossible in the DMControl suite.
As such, only the relative positioning of CIs should be considered..
Additionally, Figure 2 shows IQM and 95% CIs as a function of environment
steps (right). Both of these results show that HKSL significantly outperforms
our baselines across our 30 environment testing suite. See Appendix D for
individual environment results. We note that simply using a forward model does
not guarantee improved performance, as suggested by the comparison between DrQ
and PI-SAC.

Figure 2: IQM (left) and optimality gap (middle) of evaluation returns at 100k
environment steps, and IQM throughout training (right) across all 30 tasks.
Shaded areas are 95% confidence intervals.

###  5.3 Component Ablations

We probe each component of HKSL to determine its contribution to the overall
RL policy learning process. Specifically, we test SAC without the hierarchical
model but with HKSL’s ensemble of critics (No Repr), HKSL where each level in
the hierarchy moves with a single step (All $n=1$), HKSL without $c$ (No $c$),
HKSL where each level in the hierarchy shares encoders (Shared Encoder), and
single-level HKSL ($h=1$). The No Repr ablation tests whether HKSL’s
performance boost is due to the ensemble of critics or the hierarchical model
itself. The All $n=1$ ablation tests our hypothesis that only learning
representations at the environment’s presented temporal coarseness can miss
out on important information. The No $c$ ablation tests the value of sharing
information between levels. The Shared Encoder ablation tests if level
specialization is necessary or if one encoder can learn information at varying
temporal coarseness. Finally, the $h=1$ ablation tests the value of the
hierarchy itself by using a standard forward model (e.g., [23, 47]).

See Figure 3 for the performance comparison between these ablations and full
HKSL in the no distractors setting of Cartpole, Swingup. All results are
reported as IQMs and 95% CIs over five seeds. We highlight that variations
without all components perform worse than full HKSL. This suggests that HKSL
requires each of the individual components to achieve its full potential.

Figure 3: IQM of evaluation returns and 95% CIs for HKSL ablations.

###  5.4 Exploring HKSL Specialization and $\mathbf{c}$

Can HKSL’s levels learn to specialize? We hypothesize that one factor leading
to HKSL’s sample efficiency is its ability to divide attention to
environmental components that move at different speeds. To this end, we test a
two-level hierarchical model in Ball in Cup, Catch (BiC) with skips of
$n^{1}=1$ and $n^{2}=3$. BiC is an interesting environment for testing our
hypothesis because its two core components (the ball and the cup) have
significantly different movement characteristics in response to an agent’s
action. For example, the agent controls the cup directly, and therefore the
effects of an action on the cup manifest immediately. In contrast, the ball is
connected to the cup with a soft body (a string), and therefore its response
to agent control takes multiple timesteps to manifest. Here, we hypothesize
that coarser and finer levels should attend to the ball’s and cup’s dynamics,
respectively.

To test our hypothesis, we collect 100 episodes of observations and the
corresponding ground-truth locations of the ball and the cup with a random
policy. Then, we use the trained encoders of the two-level model to embed the
observations and fit a set of linear models to predict the ground-truth
locations from the representations. Finally, we step through an episode using
the forward models with random actions. We use the trained linear models to
predict the location of the ball and cup at every step with the latent
representations produced by the forward models. We measure the $\ell_{2}$
distance between the predicted locations and the ground-truth locations.
Figure 4 shows the prediction errors for the ball (left) and the cup (right)
over 50 episodes. We note that the forward model that moves at a coarser
temporal abstraction ($l=2$) captures the ball’s dynamics more accurately than
the forward model that moves at a finer temporal abstraction ($l=1$). These
results suggest that levels can learn to specialize.

Additionally, Figure 6 visualizes the the predicted and ground-truth locations
for the ball (top) and cup (bottom) over a trajectory of randomly-chosen
actions. These timelines show that $l=2$’s forward model produces
representations that capture the ball’s information more precisely than
$l=1$’s forward model and vice versa for the cup.

Figure 4: Prediction error of the linear models on the ball’s (left) and the
cup’s (right) position. Mean (bold line) $\pm$ one standard deviation (shaded
area) over fifty episodes.

Figure 6: Visualization of the ball-location (top) and cup-location (bottom)
predictions from the fitted linear models for $l=2$ (blue) and $l=1$ (red) as
a function of HKSL’s forward models. The observations occur chronologically
left-to-right with skips of 3 environment steps.

What does $c$ consider? We hypothesize that the communication manager
$c^{l,l-1}$ provides a wide diversity of information for $f^{l-1}$ by taking
into account the current transition of the below level $l-1$ as well as the
representations from the above level $l$. To check this hypothesis, we perform
two tests. First, we measure the $\ell_{2}$ distance between the vectors
produced by $c$ when the step $t$ is changed and other inputs are held fixed.
If $c$ completely ignores $t$, the distance between $c(\cdot,1)$ and
$c(\cdot,4)$, for example, would be zero. Second, we examine the separability
of $c$’s outputs on a trajectory-wise basis. If two sampled trajectories are
very different, then the representations produced by the above level should
change $c$’s output such that either trajectory should be clearly separable.

We first train an HKSL agent with two levels in Cartpole, Swingup for 100k
environment steps and collect 50 episodes of experiences with a random policy.
Then, we randomly sample a trajectory from this collection and step through
the latent space with both forward models. We repeat this 100 times, capture
$c$’s output, and measure the pairwise $\ell_{2}$ distance between every value
of $t$ within each sampled trajectory. Figure 7 (left) reports the average
distance between each pair. We note that the distance between $c$’s output
grows as the steps between the pairs grows. This suggests that $c$ considers
the transition of the level below it when deciding what information to share.
Additionally, we highlight that the distance increases consistently where
pairs that are the same number of steps apart are about the same distance
apart. For example, pairs $(2,5)$ and $(3,6)$ are both three steps apart and
share roughly the same average $\ell_{2}$ distance. This suggests that $c$
produces representations that are grouped smoothly in the latent space. Figure
7 (right) visualizes the PCA projections of $c$’s outputs from 20 randomly
sampled trajectories, where each trajectory is a different color. This figure
confirms our second intuition, as the representations are clearly separable on
a trajectory-wise basis with representations smoothly varying across steps
within the same trajectory.

Figure 7: Average distance between vectors produced by $c$ (left). The numbers
along the side and bottom correspond to the value of $t$. PCA projections of
representations produced by $c$ for multiple timesteps across 20 trajectories
(right) with colors corresponding to trajectories.

##  6 Limitations and Future Work

While HSKL has better sample efficiency than our baselines, its main
limitation is that its update steps are computationally expensive. The compute
expense is due to the loops over each step in the sampled trajectories and the
loops over each level in the hierarchy. This wall-clock-time challenge makes
it difficult to perform an extensive hyperparameter search. See Appendix E for
a comparison of compute times between methods. A valuable direction for future
research would be to develop methods that automatically identify the most
useful magnitudes of temporal coarseness before or during agent-environment
interaction.

##  7 Conclusion

This paper presented Hierarchical $k$-Step Latent (HKSL), an auxiliary task
for accelerating control learning from pixels via a hierarchical latent
forward model. Our experiments showed that HKSL’s representations can
substantially improve the sample efficiency of downstream RL agents in pixel-
based robotic control tasks. We also showed that the temporal coarseness of
levels in the hierarchy can lead to specialization on objects whose response
to agent actions manifests at different speeds. Finally, we showed that the
communication manager organizes information in response to the above and below
levels.

## References

  * Kalashnikov et al. [2018] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine.  QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  In _2nd Conference on Robot Learning (CoRL)_ , 2018.
  * Kalashnikov et al. [2021] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman.  Scaling up multi-task robotic reinforcement learning.  In _Proceedings of the 5th Conference on Robot Learning (CoRL)_ , 2021\.
  * Lu et al. [2021] Y. Lu, K. Hausman, Y. Chebotar, M. Yan, E. Jang, A. Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, and S. Levine.  Aw-opt: Learning robotic skills with imitation and reinforcement at scale.  In _roceedings of the 5th Conference on Robot Learning (CoRL)_ , 2021\.
  * Chebotar et al. [2021] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. C. Julian, and C. F. andyou Sergey Levine.  Actionable models: Unsupervised offline reinforcement learning of robotic skills.  In _Proceedings of the 38th International Conference on Machine Learning (ICML)_ , 2021.
  * Yarats et al. [2020] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus.  Improving sample efficiency in model-free reinforcement learning from images.  _arXiv preprint arXiv:1910.01741_ , 2020.
  * Jaderberg et al. [2017] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  In _International Conference on Learning Representations_ , 2017.
  * Laskin et al. [2020] M. Laskin, A. Srinivas, and P. Abbeel.  CURL: Contrastive unsupervised representations for reinforcement learning.  In _Proceedings of the 37th International Conference on Machine Learning (ICML)_ , volume 119, pages 5639–5650, 2020.
  * Stooke et al. [2021] A. Stooke, K. Lee, P. Abbeel, and M. Laskin.  Decoupling representation learning from reinforcement learning.  In _Proceedings of the 38th International Conference on Machine Learning (ICML)_ , volume 139, pages 9870–9879, 2021.
  * Yarats et al. [2021] D. Yarats, I. Kostrikov, and R. Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Laskin et al. [2020] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas.  Reinforcement learning with augmented data.  In _34th Conference on Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 19884–19895, 2020.
  * Lee et al. [2020] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 741–752, 2020.
  * Zhang et al. [2021] A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. Levine.  Learning invariant representations for reinforcement learning without reconstruction.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Gelada et al. [2019] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare.  DeepMDP: Learning continuous latent space models for representation learning.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , 2019.
  * Hafner et al. [2020] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi.  Dream to control: Learning behaviors by latent imagination.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Hafner et al. [2019] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International Conference on Machine Learning (ICML)_ , pages 2555–2565, 2019.
  * Tassa et al. [2018] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller.  DeepMind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Stone et al. [2021] A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski.  The distracting control suite – a challenging benchmark for reinforcement learning from pixels.  _arXiv preprint arXiv:2101.02722_ , 2021.
  * Bellman [1957] R. Bellman.  A markovian decision process.  _Indiana University Mathematics Journal_ , 6:679–684, 1957\.
  * Kaelbling et al. [1998] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra.  Planning and acting in partially observable stochastic domains.  _Artificial Intelligence_ , 101(1):99–134, 1998\.
  * Lee et al. [2020] K.-H. Lee, I. Fischer, A. Liu, Y. Guo, H. Lee, J. Canny, and S. Guadarrama.  Predictive information accelerates learning in rl.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 11890–11901, 2020.
  * Burda et al. [2019] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros.  Large-scale study of curiosity-driven learning.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International Conference on Machine Learning (ICML)_ , 2017.
  * Schwarzer et al. [2021] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman.  Data-efficient reinforcement learning with self-predictive representations.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Ha and Schmidhuber [2018] D. Ha and J. Schmidhuber.  World models.  _arXiv preprint arXiv:1803.10122_ , 2018.
  * Zhang et al. [2019] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine.  Solar: Deep structured representations for model-based reinforcement learning.  In _International Conference on Machine Learning (ICML)_ , 2019.
  * Schmidhuber [1991] J. Schmidhuber.  Neural sequence chunkers.  Technical report, 1991.
  * Chung et al. [2017] J. Chung, S. Ahn, and Y. Bengio.  Hierarchical multiscale recurrent neural networks.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Kim et al. [2019] T. Kim, S. Ahn, and Y. Bengio.  Variational temporal abstraction.  In _33rd Conference on Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Saxena et al. [2021] V. Saxena, J. Ba, and D. Hafner.  Clockwork variational autoencoders.  In _35th Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Kenter et al. [2019] T. Kenter, V. Wan, C.-A. Chan, R. Clark, and J. Vit.  CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , 2019.
  * Chen et al. [2018] H. Chen, B. Perozzi, Y. Hu, and S. Skiena.  Harp: Hierarchical representation learning for networks.  In _The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)_ , 2018.
  * Steccanella et al. [2021] L. Steccanella, S. Totaro, and A. Jonsson.  Hierarchical representation learning for markov decision processes.  _arXiv preprint: arXiv:2106.01655_ , 2021.
  * Koutnik et al. [2014] J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber.  A clockwork rnn.  In _Proceedings of the 31st International Conference on Machine Learning (ICML)_ , 2014.
  * Kumar et al. [2020] M. Kumar, M. Babaeizadeh, D. Erhan, C. Finn, S. Levine, L. Dinh, and D. Kingma.  Videoflow: A conditional flow-based model for stochastic video generation.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Castrejon et al. [2019] L. Castrejon, N. Ballas, and A. Courville.  Improved conditional vrnns for video prediction.  In _International Conference on Computer Vision (ICCV)_ , 2019.
  * He et al. [2020] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9726–9735, 2020.
  * Cho et al. [2014] K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio.  On the properties of neural machine translation: Encoder–decoder approaches.  In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , pages 103–111, 2014.
  * Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko.  Bootstrap your own latent: A new approach to self-supervised learning.  In _34th Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Tarvainen and Valpola [2017] A. Tarvainen and H. Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _31st Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * van den Oord et al. [2018] A. van den Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Fischer [2020] I. Fischer.  The conditional entropy bottleneck.  _Entropy_ , 2020.
  * Haarnoja et al. [2018a] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning (ICML)_ , volume 80, pages 1861–1870, 2018a.
  * Haarnoja et al. [2018b] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033, 2012.
  * Tassa et al. [2020] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess.  dm_control: Software and tasks for continuous control.  _arXiv preprint arXiv:2006.12983_ , 2020.
  * Agarwal et al. [2021] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  2021\.
  * McInroe et al. [2021] T. McInroe, L. Schäfer, and S. V. Albrecht.  Learning temporally-consistent representations for data-efficient reinforcement learning.  _arXiv preprint: arXiv:2110.04935_ , 2021.
  * Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Singh and Krishnan [2020] S. Singh and S. Krishnan.  Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Islam et al. [2017] R. Islam, P. Henderson, M. Gomrokchi, and D. Precup.  Reproducibility of benchmarked deep reinforcement learning tasks for continuous control.  In _Proceedings of the ICML 2017 workshop on Reproducibility in Machine Learning (RML)_ , 2017.
  * Henderson et al. [2018] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger.  Deep reinforcement learning that matters.  In _Proceedings of The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)_ , 2018.

##  Appendix A Extended Background

Soft Actor-Critic. Soft Actor-Critic (SAC) [42, 43] is a popular off-policy,
model-free RL algorithm for continuous control. SAC uses a state-action value-
function critic $Q$ and target critic $\bar{Q}$, a stochastic actor $\pi$, and
a learnable temperature $\alpha$ that weighs between reward and entropy:
$\mathbb{E}_{o_{t},a_{t}\sim\pi}[\sum_{t}\mathcal{R}(o_{t},a_{t})+\alpha\mathcal{H}(\pi(\cdot|o_{t}))]$.

SAC’s critic is updated with the squared Bellman error over historical
trajectories ${\tau=(o_{t},a_{t},r_{t},o_{t+1})}$ sampled from a replay memory
$\mathcal{D}$:

$\mathcal{L}_{critic}=\mathbb{E}_{\tau\sim\mathcal{D}}[(Q(o_{t},a_{t})-(r_{t}+\gamma
y))^{2}],$ (3)

where $y$ is computed by sampling the current policy:

$y=\mathbb{E}_{a^{\prime}\sim\pi}[\bar{Q}(o_{t+1},a^{\prime})-\alpha\log\pi(a^{\prime}|o_{t+1})].$
(4)

The target critic $\bar{Q}$ does not receive gradients, but is updated as an
exponential moving average (EMA) of $Q$ (e.g., He et al. [36]). SAC’s actor
parameterizes a multivariate Gaussian $\mathcal{N}(\mu,\sigma)$ where $\mu$ is
a vector of means and $\sigma$ is the diagonal of the covariance matrix. The
actor is updated via minimizing :

$\mathcal{L}_{actor}=-\mathbb{E}_{a\sim\pi,\tau\sim\mathcal{D}}[Q(o_{t},a)-\alpha\log\pi(a|o_{t})],$
(5)

and $\alpha$ is learned against a static value.

##  Appendix B Environments

Table 1 outlines the action space, the action repeat hyperparameter, and the
reward function type of each environment used in this study. The action repeat
hyperparameters that are displayed in the table are the standards as defined
by [15] and are the same used in most studies in DMControl. The versions of
each environment with distractors follow the presented information as well.

Table 1: Dimensions of action spaces, action repeat values, and reward function type for all six environments in the PlaNet benchmark suite. Environment, Task | $dim(\mathcal{A})$ | Action Repeat | Reward Type
---|---|---|---
Finger, spin | 2 | 2 | Dense
Cartpole, swingup | 1 | 8 | Dense
Reacher, easy | 2 | 4 | Sparse
Cheetah, run | 6 | 4 | Dense
Walker, walk | 6 | 2 | Dense
Ball in Cup, catch | 2 | 4 | Sparse

##  Appendix C Architecture and Hyperparameters

###  C.1 SAC Settings

All encoders follow the same architecture as defined by [5]. These encoders
are made of four convolutional layers separated by ReLU nonlinearities, a
linear layer with 50 hidden units, and a final layer norm opertion [48]. Each
convolutional layer has 32 3$\times$3 kernels and the layers have a stride of
2, 1, 1, and 1, respectively. This in contrast to the encoder used in the PI-
SAC study [20], which uses Filter Response Normalization [49] layers between
each convolution.

The architectures used by the SAC networks follow the same architecture as
deinfed by [5]. Both the actor and critic networks have two layers with 1024
hidden units, separated by ReLU nonlinearities. This is in contrast to the
networks used in the PI-SAC study, which uses a different number of hidden
units in the actor and critic networks.

Several studies have shown that even small differences in neural network
architecture can cause statistically signficant differences in performance
[50, 51]. As such, we avoid using the original PI-SAC encoder and SAC
architectures to ensure a fair study between all methods.

Table 2 shows the SAC hyperparameters used by all methods in this study. For
method-specific hyperparameters (e.g., auxiliary learning rate, architexture
of auxiliary networks, etc.), we defaulted to the settings provided by the
original authors.

Table 2: SAC Hyperparameters used to produce paper’s main results. Hyperparameter | Value
---|---
Image padding | 4 pixels
Initial steps | 1000
Stacked frames | 3
Evaluation episodes | 10
Optimizer | Adam
$(\beta_{1},\beta_{2})$ Optimizer | $(0.9,\;0.999)$
Learning rate | $1e-3$
Batch size | 128
Q function EMA | 0.01
Encoder EMA | 0.05
Target critic update freq | 2
$dim(z)$ | 50
$\gamma$ | 0.99
Initial $\alpha$ | 0.1
Target $\alpha$ | \- $\lvert\mathcal{A}\rvert$
Replay memory capacity | 100,000
Actor log stddev bounds | [-10,2]

###  C.2 HKSL Hyperparameters

Table 3 shows the hyperparameters that control HKSL. $h$ represents the number
of levels, $n$ contains a list of the skips of each level from lowest to
highest level, $k$ shows the length of the trajectory sampled at each training
step, learning rate corresponds to the learning rate of all HKSL’s components,
and actor update freq corresponds to the number of steps between each actor
update. These hyperparameters were found with a brief search over the non-
distractor setting of each environment.

HKSL’s communication manager $c$ is a simple two-layer non-linear model. The
first layer has 128 hidden units and the second has 50. The two layers are
separated by a ReLU nonlinearity.

Table 3: Hyperparameters used for HKSL for each environment. Environment, Task | $h$ | $n$ | $k$ | Learning rate | Actor Update Freq
---|---|---|---|---|---
Finger, spin | 2 | [1,3] | 3 | 1e-4 | 2
Cartpole, swingup | 2 | [1,3] | 6 | 1e-3 | 1
Reacher, easy | 2 | [1,3] | 3 | 1e-4 | 2
Cheetah, run | 2 | [4,5] | 10 | 1e-4 | 2
Walker, walk | 2 | [1,3] | 6 | 1e-3 | 1
Ball in Cup, catch | 2 | [1,3] | 6 | 1e-3 | 1

###  C.3 HKSL’s Forward Models

The usual GRU formulation at step $t$:

$\displaystyle u^{t}_{gru}=\sigma(f^{u}_{gru}([a_{t}|z_{t-1}]))$ (6)
$\displaystyle r^{t}_{gru}=\sigma(f^{r}_{gru}([a_{t}|z_{t-1}]))$ (7)
$\displaystyle h^{t}_{gru}=tanh(f^{h}_{gru}([r^{t}_{gru}\odot
z_{t-1}|a_{t}]))$ (8) $\displaystyle g^{t}_{gru}=(1-u^{t}_{gru})\odot
z_{t-1}+u^{t}_{gru}\odot h^{t}_{gru}$ (9)

where each each distinct $f$ is an affine transform, $\sigma$ is the sigmoid
nonlinearity, and $\odot$ is the Hadamard product. In order to allow the
forward models to take the optional input from $c$, we add an identical set of
additional affine transforms:

$\displaystyle u^{t}_{c}=\sigma(f^{u}_{c}([C_{t}|z_{t-1}]))$ (10)
$\displaystyle r^{t}_{c}=\sigma(f^{r}_{c}([C_{t}|z_{t-1}]))$ (11)
$\displaystyle h^{t}_{c}=tanh(f^{h}_{c}([r^{t}_{c}\odot C_{t}|z_{t-1}]))$ (12)
$\displaystyle g^{t}_{c}=(1-u^{t}_{c})\odot z_{t-1}+u^{t}_{c}\odot h^{t}_{c}$
(13)

where $C_{t}$ denotes the output from $c$ at step $t$. Finally, the output of
the forward model is the average of the two pathways:

$z_{t}=\frac{g^{t}_{c}+g^{t}_{gru}}{2}$ (14)

##  Appendix D Individual Environment Results

This section shows the mean (bold lines) $\pm$ one standard deviation (shaded
area) for every individual environment and distractor combination. Figure 8
displays the non-distractor environments, Figure 9 shows the color distractors
on the easy setting, Figure 10 shows the color distractors on the medium
setting, Figure 11 shows the camera distractors on the easy settings, and
Figure 12 shows the camera distractors on the medium setting.

Figure 8: Evaluation returns for agents trained in DMControl without
distractors. Bold line depicts the mean and shaded area represents $+-$ one
standard deviation across five seeds. Figure 9: Evaluation returns for agents
trained in DMControl with color distractors on the easy setting. Bold line
depicts the mean and shaded area represents $+-$ one standard deviation across
five seeds. Figure 10: Evaluation returns for agents trained in DMControl with
color distractors on the medium setting. Bold line depicts the mean and shaded
area represents $+-$ one standard deviation across five seeds. Figure 11:
Evaluation returns for agents trained in DMControl with camera distractors on
the easy setting. Bold line depicts the mean and shaded area represents $+-$
one standard deviation across five seeds. Figure 12: Evaluation returns for
agents trained in DMControl with camera distractors on the medium setting.
Bold line depicts the mean and shaded area represents $+-$ one standard
deviation across five seeds.

##  Appendix E Compute Expense

We measure the wall-clock time required to train the methods used in this
paper. To do so, we measure the wall-clock time required to train for 10k
environment steps in the no distractor setting of Cartpole, Swingup. Table 4
shows each method normalized to the time of base SAC with no image
augmentations. For HKSL’s settings, see Section C.2. We note that the value of
HKSL’s normalized compute time will change based on its chosen
hyperparameters.

Table 4: Training time of agent methods, normalized to base SAC. Method | Normalized Time
---|---
SAC | 1.0
HKSL | 40.0
DrQ | 1.8
CURL | 1.6
PI-SAC | 2.7


# Quantum correlations and ergotropy

Gianluca Francica<EMAIL_ADDRESS>Dipartimento di Fisica e
Astronomia “G. Galilei”, Università degli Studi di Padova, via Marzolo 8,
35131 Padova, Italy

###### Abstract

Understanding the role of classical and quantum correlations in work
extraction is a problem of fundamental importance in thermodynamics. We
approach this problem by considering that, in closed quantum systems, the
maximum cyclic work extractable is equal to the ergotropy. Thus, we aim to
identify and investigate the contributions to the ergotropy coming from
different kinds of initial correlations (total, classical, discord and
entanglement correlations). By doing so, we have introduced and studied
quantifiers of correlations which are based on ergotropy. In particular, our
results suggest that only discord correlations always give a positive
contribution to work extraction, while total, classical and entanglement
correlations can reduce the work extraction.

Introduction. Quantum correlations like entanglement einstein35 ;
schrodinger35 ; bohr35  are possible thanks to the tensor product structure of
the Hilbert space of a multipartite system and the superposition principle.
These correlations cannot be generated from an uncorrelated state by using
only local quantum operations and classical communication (see, e.g., Ref.
horodecki09  for a review). However, there exist quantum correlations,
quantified by the quantum discord henderson01 ; ollivier01 , which are due to
quantum physical effects but do not necessarily involve entanglement. In
general, quantum correlations have received much attention in studies
involving thermodynamics (see, e.g., Ref. book19 ). Concerning the work
extraction in closed quantum systems, the ergotropy allahverdyan04 , defined
as the maximum work extractable by using a cyclic unitary transformation, is a
key quantity of fundamental importance. From this point of view, similarly to
the work deficit oppenheim02 , the difference between the global and total
local ergotropy is called ergotropic gap perarnau-llobet15 ; mukherjee16 ,
which is shown to be related to the presence of entanglement alimuddin19 .
Moreover, the transformation of equal energetic states under the unital
operation has also been studied alimuddin20 . Furthermore, by considering an
ancilla-assisted protocol, the quantum correlations are related to a possible
increase of the extracted work (daemonic ergotropy), acquiring a so-called
daemonic gain francica17 . Regarding quantum coherence, recently, the role of
the initial quantum coherence in the energy basis has also been investigated
francica20 ; francica22 . Here, we adopt the same perspective of Ref.
francica20 , where the initial quantum coherence contribution to the ergotropy
is identified, and we aim to clarify the role of initial correlations as a
resource for work extraction. For simplicity, we will focus our discussion on
bipartite systems, but our results are easily applicable for multipartite
systems.

Preliminaries. Given a quantum system of two parties $A$ and $B$, with Hilbert
space $\mathcal{H}=\mathcal{H}_{A}\otimes\mathcal{H}_{B}$, we are interested
in the correlations of an initial state $\rho$. Let us distinguish the
correlations by following Ref. modi10 . We start by considering a state
without correlations, which is a product state $\pi=\pi_{A}\otimes\pi_{B}$,
where $\pi_{A}$ ($\pi_{B}$) is the reduced state of the subsystem $A$ ($B$).
By considering the set $\mathcal{P}$ of all product states, the total
correlations are quantified by $T(\rho)=\min_{\pi\in\mathcal{P}}S(\rho||\pi)$,
where we have defined the quantum relative entropy
$S(\rho||\eta)=\text{Tr}\left\\{\rho(\ln\rho-\ln\eta)\right\\}$, which is non-
negative and $S(\rho||\eta)=0$ if and only if $\rho=\eta$. We have that
$T(\rho)=S(\rho||\pi_{\rho})$, where $\pi_{\rho}=\rho_{A}\otimes\rho_{B}$,
then it is equal to the mutual information
$T(\rho)=S(\rho_{A})+S(\rho_{B})-S(\rho)$, where we have defined the von
Neumann entropy $S(\rho)=-\text{Tr}\left\\{\rho\ln\rho\right\\}$. Conversely,
a state which is a mixture of locally distinguishable states is a classical
state $\chi=\sum_{i,j}p_{ij}P^{A}_{i}\otimes P^{B}_{j}$, where $p_{ij}$ is a
joint probability distribution and $P^{A}_{i}$ and $P^{B}_{j}$ form a complete
set of orthogonal projectors of rank one. The correlations of these states are
identified as classical correlations, which are $C(\chi)=T(\chi)$. By
considering the set $\mathcal{C}$ of all the classical states, the amount of
quantum correlations can be quantified by the relative entropy of discord
$D(\rho)=\min_{\chi\in\mathcal{C}}S(\rho||\chi)=S(\rho||\chi_{\rho})$, where
$\chi_{\rho}$ is the closest classical state. Thus, the mutual information can
be expressed as $T(\rho)=D(\rho)+C(\chi_{\rho})-L(\rho)$, which defines the
quantity $L(\rho)$. In particular, by substituting in the equation all the
definitions given, it is easy to see that
$L(\rho)=S(\pi_{\rho}||\pi_{\chi_{\rho}})$ 111Since $\chi_{\rho}$ is obtained
by applying a dephasing map to $\rho$, as we will notice later, we get
$\text{Tr}\left\\{\rho\ln\chi_{\rho}\right\\}=\text{Tr}\left\\{\chi_{\rho}\ln\chi_{\rho}\right\\}$,
which implies $D(\rho)=S(\chi_{\rho})-S(\rho)$, and similarly
$L(\rho)=S(\pi_{\chi_{\rho}})-S(\pi_{\rho})=S(\pi_{\rho}||\pi_{\chi_{\rho}})$..
Finally, a state which can be prepared using only local quantum operations and
classical communication is a separable state
$\sigma=\sum_{i}p_{i}\pi^{(i)}_{A}\otimes\pi^{(i)}_{B}$, where $p_{i}$ is a
probability and $\pi^{(i)}_{A}$ and $\pi^{(i)}_{B}$ are local states of the
subsystems, and we indicate with $\mathcal{S}$ the set of all separable
states. We note that $\mathcal{P}\subset\mathcal{C}\subset\mathcal{S}$ and
while $\mathcal{S}$ is convex, $\mathcal{P}$ and $\mathcal{C}$ are not. If
$\rho$ is not separable then it is entangled and the entanglement can be
quantified by the relative entropy of entanglement
$E_{re}(\rho)=\min_{\sigma\in\mathcal{S}}S(\rho||\sigma)=S(\rho||\sigma_{\rho})$,
where $\sigma_{\rho}$ is the closest separable state. In particular, separable
states can possess nonclassical features and $D(\sigma_{\rho})$ is called
quantum dissonance. About the discord, it is shown that the closest classical
state can be expressed as $\chi_{\rho}=\sum_{i,j}P^{A}_{i}\otimes
P^{B}_{j}\rho P^{A}_{i}\otimes P^{B}_{j}$, then
$D(\rho)=\min_{\\{P^{A}_{i},P^{B}_{j}\\}}S(\chi_{\rho})-S(\rho)$, since
$\text{Tr}\left\\{\rho\ln\chi_{\rho}\right\\}=\text{Tr}\left\\{\chi_{\rho}\ln\chi_{\rho}\right\\}$.
In order to avoid the minimization procedure, the measurement induced
disturbance has been introduced as
$D^{\prime}(\rho)=T(\rho)-T(\chi^{\prime}_{\rho})=S(\chi^{\prime}_{\rho})-S(\rho)$
where $\chi^{\prime}_{\rho}$ is achieved by considering projectors
corresponding to the marginal basis of the eigenstates of the reduced states
$\rho_{A}$ and $\rho_{B}$, and of course
$\pi_{\chi^{\prime}_{\rho}}=\pi_{\rho}$. Thus, we get
$T(\rho)=D^{\prime}(\rho)+C(\chi^{\prime}_{\rho})$ and, if $L(\rho)=0$,
$D^{\prime}(\rho)=D(\rho)$. For a pure state $\rho=|\psi\rangle\langle\psi|$,
if we consider the Schmidt decomposition
$|\psi\rangle=\sum_{i}\sqrt{p_{i}}|i_{A}\rangle\otimes|i_{B}\rangle$, where
$\sqrt{p_{i}}$ are called Schmidt coefficients, the closest classical state is
$\chi_{\rho}=\sum_{i}p_{i}|i_{A}\rangle\langle
i_{A}|\otimes|i_{B}\rangle\langle i_{B}|$ and
$D(\rho)=S(\chi_{\rho})=-\sum_{i}p_{i}\ln p_{i}$. In this case, we get also
$\sigma_{\rho}=\chi_{\rho}$ and $E_{re}(\rho)=D(\rho)$, then entanglement is
the only quantum correlation. However, things are different for a pure
multipartite state, where the dissonance can be non-zero.

Following Ref. allahverdyan04 , we are now interested in extracting work from
the quantum system in the initial state $\rho$ by using a cyclic unitary
transformation, i.e., by considering a unitary time evolution operator $U$
generated by a time-dependent Hamiltonian which at the final time is equal to
the initial Hamiltonian $H$. Since the system is thermally isolated, the
average work $W$ extracted from the system is equal to minus the change of
average energy, i.e., $W=E(\rho)-E(U\rho U^{\dagger})$, where the average
energy is defined as $E(\rho)=\text{Tr}\left\\{H\rho\right\\}$. The maximum of
$W$ over the set of all the unitary operators $U$ is called ergotropy,
$\mathcal{E}(\rho)$. After ordering the labels of eigenstates of $H$ and of
$\rho$ such that
$H=\sum_{k}\epsilon_{k}|\epsilon_{k}\rangle\langle\epsilon_{k}|$, with
$\epsilon_{k}\leq\epsilon_{k+1}$, and $\rho=\sum_{k}r_{k}|r_{k}\rangle\langle
r_{k}|$, with $r_{k}\geq r_{k+1}$, the ergotropy reads
$\mathcal{E}(\rho)=E(\rho)-E(P_{\rho})$, where $P_{\rho}$ is the passive state
$P_{\rho}=\sum_{k}r_{k}|\epsilon_{k}\rangle\langle\epsilon_{k}|$, which for
instance can be obtained as $P_{\rho}=U\rho U^{\dagger}$ with
$U=\sum_{k}|\epsilon_{k}\rangle\langle r_{k}|$. The ergotropy is non-negative
and $\mathcal{E}(\rho)=0$ if and only if $\rho$ is passive. We can compare the
ergotropy of two states $\rho$ and $\eta$. If the two states have the same
energy, $E(\rho)=E(\eta)$, and there exists a unital channel (i.e., which maps
the identity to the identity) $\Lambda$ such that $\eta=\Lambda(\rho)$, we
have $\mathcal{E}(\rho)\geq\mathcal{E}(\eta)$ allahverdyan04 . To prove it, we
recall that $\eta=\Lambda(\rho)$ if and only if $\eta\prec\rho$ if and only if
there exist unitary operators $U_{i}$ and probabilities $p_{i}$ such that
$\eta=\sum_{i}p_{i}U_{i}\rho U_{i}^{\dagger}$ nielsen01 . We note that
$E(P_{\rho})$ is a concave function of $\rho$, then
$E(P_{\eta})\geq\sum_{i}p_{i}E(P_{U_{i}\rho U_{i}^{\dagger}})=E(P_{\rho})$
since $P_{\rho}=P_{U\rho U^{\dagger}}$ for any unitary $U$, thus we get
$\mathcal{E}(\rho)-\mathcal{E}(\eta)=E(P_{\eta})-E(P_{\rho})\geq 0$. In
general, for two states $\rho$ and $\eta$ having the same energy, it is easy
to show the identity

$\beta(\mathcal{E}(\rho)-\mathcal{E}(\eta))=S(\eta)-S(\rho)-S(P_{\rho}||\rho_{\beta})+S(P_{\eta}||\rho_{\beta})\,$
(1)

for any thermal state $\rho_{\beta}=e^{-\beta H}/\text{Tr}\left\\{e^{-\beta
H}\right\\}$.

Thus, by considering a multipartite scenario, our aim is to demonstrate a
precise connection between the ergotropy $\mathcal{E}(\rho)$ and the
correlations in the initial state $\rho$. In order to do this, we proceed in a
similar way as for quantifying correlations, by searching the corresponding
closest states, i.e., the states with same energy of $\rho$ and without the
desired property (e.g., mutual information, entanglement or discord) which
maximize the probability of being not distinguishable from $\rho$, and so
minimize the quantum relative entropy vedral97 .

Correlations contributions to the ergotropy. If initially the two parties do
not interact with each other, the initial Hamiltonian is $H=H_{A}+H_{B}$,
where $H_{A}$ ($H_{B}$) is the Hamiltonian of the part $A$ ($B$). We mainly
focus on this case, so that $E(\pi_{\rho})=E(\rho)$ and $\pi_{\rho}$ is the
closest product state with same energy. In analogy to the mutual information,
we can define the contribution of the total correlations $\delta_{T}(\rho)$
such that

$\mathcal{E}(\rho)=\mathcal{E}(\pi_{\rho})+\delta_{T}(\rho)\,,$ (2)

and $\delta_{T}(\pi)=0$ if $\pi$ is a product state. By using the identity of
Eq. (1), we get

$\beta\delta_{T}(\rho)=T(\rho)-S(P_{\rho}||\rho_{\beta})+S(P_{\pi_{\rho}}||\rho_{\beta})\,,$
(3)

which reduces to the expression for $\mathcal{E}(\rho)$ recently introduced in
Ref. touil22  for a local thermal initial state, $\pi_{\rho}=\rho_{\beta}$.
Since the quantum relative entropy is non-negative, we get the bounds

$T(\rho)-S(P_{\rho}||\rho_{\beta})\leq\beta\delta_{T}(\rho)\leq
T(\rho)+S(P_{\pi_{\rho}}||\rho_{\beta})$ (4)

for any $\beta$. We note that our quantity is different from the ergotropic
gap perarnau-llobet15 ; mukherjee16 , defined as
$\Delta_{EG}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\rho_{A})-\mathcal{E}(\rho_{B})$.
In particular, we get
$\Delta_{EG}(\rho)-\delta_{T}(\rho)=E(P_{\rho_{A}}\otimes
P_{\rho_{B}})-E(P_{\pi_{\rho}})\geq 0$, since $P_{\rho_{A}}\otimes
P_{\rho_{B}}=U_{A}\otimes U_{B}\pi_{\rho}U_{A}^{\dagger}\otimes
U_{B}^{\dagger}$ has an energy not smaller than the one of the passive state
$P_{\pi_{\rho}}$.

For characterizing the discord contribution we consider the set
$\mathcal{C}_{E}$ of classical states $\eta$ having the same energy of $\rho$,
$E(\eta)=E(\rho)$. We note that $\pi_{\rho}\in\mathcal{C}_{E}$, so that the
set $\mathcal{C}_{E}$ is not empty. The closest classical state
$\eta_{\rho}\in\mathcal{C}_{E}$ is such that
$\min_{\eta\in\mathcal{C}_{E}}S(\rho||\eta)=S(\rho||\eta_{\rho})$. Therefore,
we define the contribution of discord correlations as

$\delta(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\eta_{\rho})\,.$ (5)

The total correlations and the discord contributions are related by

$\delta_{T}(\rho)=\delta(\rho)+\delta_{C}(\eta_{\rho})-\delta_{L}(\rho)\,,$
(6)

where, given an initial classical state $\chi$, we have defined the classical
correlations contribution as
$\delta_{C}(\chi)=\delta_{T}(\chi)=\mathcal{E}(\chi)-\mathcal{E}(\pi_{\chi})$,
and $\delta_{L}(\rho)=\mathcal{E}(\pi_{\rho})-\mathcal{E}(\pi_{\eta_{\rho}})$.
We note that these definitions of $\delta_{T}(\rho)$ and $\delta(\rho)$ are
analogous to the definition for the quantum coherence contribution to the
ergotropy francica20 ,
$\mathcal{E}_{c}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\Delta(\rho))$, where
$\Delta$ is the dephasing map with respect to the energy basis
$|\epsilon_{k}\rangle$, $\Delta(\rho)$ is the closest incoherent state to
$\rho$ and of course $E(\Delta(\rho))=E(\rho)$.

Similarly to $\chi_{\rho}$, it is easy to show that
$\eta_{\rho}=\sum_{i,j}P^{A}_{i}\otimes P^{B}_{j}\rho P^{A}_{i}\otimes
P^{B}_{j}$, where the projectors are not necessarily equal to the ones of
$\chi_{\rho}$, since in general they minimize the entropy $S(\eta_{\rho})$
with the constraint $E(\eta_{\rho})=E(\rho)$ and of course $S(\eta_{\rho})\geq
S(\chi_{\rho})$. To avoid misunderstandings, from now on we will indicate with
$P^{\prime A}_{i}$ and $P^{\prime B}_{j}$ the projectors of $\chi_{\rho}$. We
observe that $\delta(\rho)$ is invariant under local unitary transformations,
i.e., $\delta(\rho^{\prime})=\delta(\rho)$ with $\rho^{\prime}=U_{A}\otimes
U_{B}\rho U_{A}^{\dagger}\otimes U_{B}^{\dagger}$ for any unitary local
operators $U_{A}$ and $U_{B}$. This follows by noting that
$P_{\rho^{\prime}}=P_{\rho}$ and $P_{\eta_{\rho^{\prime}}}=P_{\eta_{\rho}}$
since $\eta_{\rho^{\prime}}=U_{A}\otimes
U_{B}\eta_{\rho}U_{A}^{\dagger}\otimes U_{B}^{\dagger}$. Furthermore,
$\pi_{\rho^{\prime}}=U_{A}\otimes U_{B}\pi_{\rho}U_{A}^{\dagger}\otimes
U_{B}^{\dagger}$ and thus we get $\delta_{C}(\rho^{\prime})=\delta_{C}(\rho)$
and $\delta_{L}(\rho^{\prime})=\delta_{L}(\rho)$, so also
$\delta_{T}(\rho^{\prime})=\delta_{T}(\rho)$. Since $\eta_{\rho}$ is obtained
by applying a unital channel to $\rho$, we get $\delta(\rho)\geq 0$, and of
course if $\chi$ is classical then $\delta(\chi)=0$. Therefore,
$\mathcal{E}(\rho)=\mathcal{E}(\eta_{\rho})+\delta(\rho)\geq\delta(\rho)$ and
the discord contribution $\delta(\rho)$ gives a non-negative lower bound of
the ergotropy. By using the identity of Eq. (1), we get

$\beta\delta(\rho)=D(\rho)+S(\eta_{\rho})-S(\chi_{\rho})-S(P_{\rho}||\rho_{\beta})+S(P_{\eta_{\rho}}||\rho_{\beta})\,$
(7)

from which we have the bounds

$D(\rho)-S(P_{\rho}||\rho_{\beta})\leq\beta\delta(\rho)\leq
S(\rho||\eta_{\rho})+S(P_{\eta_{\rho}}||\rho_{\beta})\,$ (8)

for any $\beta$. If $P_{\rho}$ is completely passive, i.e., is thermal,
$P_{\rho}=\rho_{\beta}$, we have $\delta_{T}(\rho)\geq T(\rho)/\beta$ and
$\delta(\rho)\geq D(\rho)/\beta$. Anyway, since the eigenvalues of $\rho$ are
conserved in the unitary transformation, the passive state $P_{\rho}$ is not
necessarily a thermal state and in this case the lower bounds can be negative
and useless. An analogous result for the classical correlations is achieved by
considering an initial state $\chi$ classical and the definition
$\delta_{C}(\chi)=\delta_{T}(\chi)$. Instead, for $\delta_{L}(\rho)$ we get

$\beta\delta_{L}(\rho)=L(\rho)+S(\pi_{\eta_{\rho}})-S(\pi_{\chi_{\rho}})-S(P_{\pi_{\rho}}||\rho_{\beta})+S(P_{\pi_{\eta_{\rho}}}||\rho_{\beta})\,.$
(9)

It is worth to observe that if the energy spectrum is non-degenerate,
$\epsilon_{k}<\epsilon_{k+1}$, then $\delta(\rho)=0$ implies that $\rho$ is
classical. On the other hand, as we will see by considering a simple example,
if the energy spectrum is degenerate we can have non-zero quantum correlations
but $\delta(\rho)=0$.

Proof. If the eigenvalues of $\eta_{\rho}$ are $q_{k}$ ordered such that
$q_{k}\geq q_{k+1}$, its passive state is
$P_{\eta_{\rho}}=\sum_{k}q_{k}|\epsilon_{k}\rangle\langle\epsilon_{k}|$ and we
get
$\delta(\rho)=\sum_{k}(\epsilon_{k+1}+\epsilon_{k})\sum_{n=1}^{k}(r_{n}-q_{n})$.
Since $\eta_{\rho}$ is obtained by applying a unital map to $\rho$, we have
that $\sum_{n=1}^{k}(r_{n}-q_{n})\geq 0$ for any $k$, thus $\delta(\rho)=0$
implies $q_{k}=r_{k}$ for any $k$ and so $P_{\eta_{\rho}}=P_{\rho}$. Then,
there exists a unitary $U$ such that $\eta_{\rho}=U\rho U^{\dagger}$, which
implies $S(\eta_{\rho})=S(\rho)$. Therefore,
$S(\rho||\eta_{\rho})=S(\eta_{\rho})-S(\rho)=0$ and so $\rho=\eta_{\rho}$.

For a pure quantum state, $\rho=|\psi\rangle\langle\psi|$, we have that
$E(\chi_{\rho})=E(\rho)$, then $\eta_{\rho}=\chi_{\rho}$. In particular,
$\delta(\rho)$ is intimately related to the entanglement and reads

$\delta(\rho)=\sum_{i}p_{i}\epsilon_{i}-\epsilon_{1}\,,$ (10)

where $p_{i}$ are the square of the Schmidt coefficients ordered such that
$p_{i}\geq p_{i+1}$. Then, if the fundamental level $\epsilon_{1}$ is non-
degenerate, $\delta(\rho)=0$ if and only if there is no entanglement. We note
that the expression of Eq. (10) differs from the daemonic gain francica17
only in energy levels.

In order to complete the discussion about the discord contribution, we
consider an illustrative physical example of two qubits. We consider
$H_{A}=R\epsilon|1\rangle\langle 1|$ and $H_{B}=\epsilon|1\rangle\langle 1|$,
where $\\{|0\rangle,|1\rangle\\}$ is the computational basis of a qubit, and
$\epsilon,R\geq 0$. We illustrate our results by focusing on the initial state
$\rho=\mu|0\rangle\langle 0|\otimes|+\rangle\langle+|+(1-\mu)|1\rangle\langle
1|\otimes|1\rangle\langle 1|$, with $\mu\in[0,1]$ and
$|\pm\rangle=(|0\rangle\pm|1\rangle)/\sqrt{2}$. The projectors of the closest
classical state $\chi_{\rho}$ are $P_{1}^{\prime A}=|0\rangle\langle 0|$,
$P_{1}^{\prime B}=|0\rangle\langle 0|$ if $\mu\leq 1/2$ and $P_{1}^{\prime
B}=|+\rangle\langle+|$ if $\mu>1/2$, and thus we get
$D(\rho)=\min\\{\mu,1-\mu\\}\ln 2$. The energy of the passive state is
$E(P_{\rho})=\min\\{1,R\\}\epsilon\min\\{\mu,1-\mu\\}$. For $\mu\leq 1/2$,
since $E(\chi_{\rho})=E(\rho)$, we get $\eta_{\rho}=\chi_{\rho}=\Delta(\rho)$,
where the dephasing is respect to the computational basis. Then,
$\delta(\rho)=\mathcal{E}_{c}(\rho)=|1-R|\epsilon\mu/2$, which is equal to
zero if $R=1$ (there is degeneracy) also if the initial state is not
incoherent and has quantum correlations. For $\mu>1/2$,
$\eta_{\rho}\neq\chi_{\rho}$ since $E(\chi_{\rho})\neq E(\rho)$. To calculate
$\eta_{\rho}$, we consider its projectors $P^{A}_{1}=|a,\alpha\rangle\langle
a,\alpha|$ and $P^{B}_{1}=|b,\beta\rangle\langle b,\beta|$, with
$|a,\alpha\rangle=\sqrt{a}|0\rangle+\sqrt{1-a}e^{i\alpha}|1\rangle$. The
condition $E(\eta_{\rho})=E(\rho)$ and $S(\eta_{\rho})$ do not depend on
$\alpha$, and we find $\beta=0$, so that $E(\eta_{\rho})=E(\rho)$ is
equivalent to
$f(a,b)=2(1-a)aR(2\mu-1)-2(1-b)b(1-\mu)+(1-2b)\sqrt{(1-b)b}\mu=0$. Thus, we
determinate the set $\mathcal{C}_{E}$ by changing $b$ and finding the
solutions of $f(a,b)=0$. Then, we find $\eta_{\rho}$ by minimizing
$S(\eta_{\rho})$ over $\mathcal{C}_{E}$. For $\mu\leq\mu_{c}(R)$, we find
$\eta_{\rho}=\Delta(\rho)$, instead for $\mu>\mu_{c}(R)$,
$\eta_{\rho}\neq\Delta(\rho)$. The contribution $\delta(\rho)$ is shown in
Fig. 1 and we can see that $\delta(\rho)$ is discontinuous at $\mu=\mu_{c}(R)$
since the set $\mathcal{C}_{E}$ is not convex.

Figure 1: Plot of $\delta(\rho)$ in unit of $\epsilon$ in function of $\mu$
for $R=1$. We find $\mu_{c}(1)\approx 0.57.$

Proceeding with the discussion of the other contributions, we note that
$\delta_{L}(\rho)\geq 0$ since $\pi_{\eta_{\rho}}=\sum_{i,j}P^{A}_{i}\otimes
P^{B}_{j}\pi_{\rho}P^{A}_{i}\otimes P^{B}_{j}$ which is a unital channel. On
the contrary, the same is not true for the classical correlations. We consider
the initial state $\chi=\sum_{i,j}p_{ij}P^{A}_{i}\otimes P^{B}_{j}$, and we
start by noting that if $P_{\chi}=U_{A}\otimes U_{B}\chi
U_{A}^{\dagger}\otimes U_{B}^{\dagger}$, the reduced states of $P_{\chi}$ are
$U_{A}\chi_{A}U_{A}^{\dagger}$ and $U_{B}\chi_{B}U_{B}^{\dagger}$. Since
$U_{A}$ and $U_{B}$ are such that the energy of $P_{\chi}$ is minimum, we have
that $P_{\chi_{A}}=U_{A}\chi_{A}U_{A}^{\dagger}$ and
$P_{\chi_{B}}=U_{B}\chi_{B}U_{B}^{\dagger}$, then
$E(P_{\chi})=E(P_{\chi_{A}}\otimes P_{\chi_{B}})\geq E(P_{\pi_{\chi}})$, from
which follows $\delta_{C}(\chi)\leq 0$. In general,
$\delta_{C}(\chi)=\sum_{k}\epsilon_{k}(\tilde{r}_{k}-r_{k})$, where $r_{k}$
and $\tilde{r}_{k}$ are the eigenvalues of $\chi$ and $\pi_{\chi}$ in non-
increasing order, $r_{k}\geq r_{k+1}$ and $\tilde{r}_{k}\geq\tilde{r}_{k+1}$.
If $d_{A}$ ($d_{B}$) is the dimension of $\mathcal{H}_{A}$
($\mathcal{H}_{B}$), for the special case $d_{A}=d_{B}=2$, we have that
$\epsilon_{4}-\epsilon_{3}=\epsilon_{2}-\epsilon_{1}$ and so
$\delta_{C}(\chi)=(\epsilon_{2}-\epsilon_{1})(r_{1}-r_{4}-\tilde{r}_{1}+\tilde{r}_{4})+(\epsilon_{3}-\epsilon_{2})(r_{1}+r_{2}-\tilde{r}_{1}-\tilde{r}_{2})$.
We find that $r_{1}-r_{4}-\tilde{r}_{1}+\tilde{r}_{4}\geq 0$ and
$r_{1}+r_{2}-\tilde{r}_{1}-\tilde{r}_{2}\geq 0$, then $\delta_{C}(\chi)\geq 0$
for any classical state $\chi$. Therefore, we have that if
$P_{\chi}=U_{A}\otimes U_{B}\chi U_{A}^{\dagger}\otimes U_{B}^{\dagger}$ or
$\chi=\pi_{\chi}$ then $\delta_{C}(\chi)=0$, but the reverse is not true. For
instance if we consider $p_{11}\geq p_{12}\geq p_{21}\geq p_{22}$ we have
$\delta_{C}(\chi)=0$, but we can have $\chi\neq\pi_{\chi}$ and $P_{\chi}\neq
U_{A}\otimes U_{B}\chi U_{A}^{\dagger}\otimes U_{B}^{\dagger}$ (if
$\epsilon^{A}_{2}<\epsilon^{B}_{2}$, where we indicate with $\epsilon^{A}_{i}$
and $\epsilon^{B}_{j}$ the eigenvalues of $H_{A}$ and $H_{B}$ in non-
decreasing order). However, for higher dimensions $\delta_{C}(\chi)$ can also
be negative, thus reduces the work extracted with respect to the product
state. In particular, $\delta_{C}(\chi)$ can be negative also if $P_{\chi}\neq
U_{A}\otimes U_{B}\chi U_{A}^{\dagger}\otimes U_{B}^{\dagger}$. For instance,
for the case $d_{A}=2$ and $d_{B}=3$, we can consider $p_{11}=0.27$,
$p_{12}=0.04$, $p_{13}=0.23$, $p_{21}=0.26$, $p_{22}=0.03$ and $p_{23}=0.17$.
We get
$\delta_{C}(\chi)=(\epsilon_{2}-\epsilon_{1})x_{1}+(\epsilon_{4}-\epsilon_{3})x_{3}+(\epsilon_{6}-\epsilon_{5})x_{5}$
with $x_{1}=-0.0162$, $x_{3}=0.014$ and $x_{5}=0.0022$, and since
$x_{3}+x_{5}\geq 0$ and $x_{1}+x_{3}+x_{5}=0$, it is easy to show that
$\delta_{C}(\rho)<0$ if $\epsilon^{A}_{1}=\epsilon^{B}_{1}=0$,
$\epsilon^{A}_{2}\leq\epsilon^{B}_{2}\leq\epsilon^{B}_{3}$ and
$\epsilon^{A}_{2}+\epsilon^{B}_{2}>\epsilon^{B}_{3}$. Anyway, a numerical
analysis suggests that the classical states $\chi$ such that
$\delta_{C}(\chi)<0$ are unlikely for large $d_{A}$ or $d_{B}$ (see Fig. 2).

Figure 2: Plot of the probability to find $\delta_{C}(\chi)<0$ in function of
the dimension $d_{B}$ for fixed $d_{A}$. We generate $\chi$ random and
$\epsilon^{A}_{i}$, $\epsilon^{B}_{j}$ random in the interval $[0,1]$ for
$n=10^{5}$ times and calculate the probability as the number of times to see
$\delta_{C}(\chi)<0$ over $n$. The probability decreases exponentially with
$d_{B}$ and tends to zero for $d_{B}\to\infty$.

Analogously, we can define the entanglement contribution by considering the
closest separable state $\tau_{\rho}$ having the same energy,
$E(\tau_{\rho})=E(\rho)$. We define

$\delta_{E}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\tau_{\rho})\,,$ (11)

so that $\delta_{E}(\sigma)=0$ if $\sigma$ is separable. Of course also
$\delta_{E}(\rho)$ is invariant under local unitary transformations. We note
that the difference
$\delta(\rho)-\delta_{E}(\rho)=\mathcal{E}(\tau_{\rho})-\mathcal{E}(\eta_{\rho})$
is related to the quantum dissonance, since it is equal to zero if
$D(\tau_{\rho})=0$. The contribution $\delta_{E}(\rho)$ can be expressed as
$\delta_{E}(\rho)=\mathcal{E}(\pi_{\rho})-\mathcal{E}(\tau_{\rho})+\delta_{T}(\rho)$
and for a pure state $\rho$, we get $\delta_{E}(\rho)=\delta(\rho)$. However,
contrary to the discord contribution, we observe that $\delta_{E}(\rho)$ can
be also negative for initial mixed states (only if $D(\tau_{\rho})\neq 0$). To
prove it, we consider $d_{A}=d_{B}=2$, the Horodecki state
$\rho=p|\psi^{(\pm)}\rangle\langle\psi^{(\pm)}|+(1-p)|00\rangle\langle 00|$,
having the closest separable state vedral98  $\sigma_{\rho}=q^{\prime
2}|00\rangle\langle
00|+2p^{\prime}q^{\prime}|\psi^{(\pm)}\rangle\langle\psi^{(\pm)}|+p^{\prime
2}|11\rangle\langle 11|$, where
$|\psi^{(\pm)}\rangle=(|01\rangle\pm|10\rangle)/\sqrt{2}$, $p^{\prime}=p/2$
and $q^{\prime}=1-p^{\prime}$. It is easy to show that
$E(\sigma_{\rho})=E(\rho)$, so that $\tau_{\rho}=\sigma_{\rho}$. Thus, if we
consider for instance $p=1/2$, $\epsilon^{A}_{1}=\epsilon^{B}_{1}=0$ and
$\epsilon^{A}_{2}=\epsilon^{B}_{2}=\epsilon$, we find
$\delta_{E}(\rho)=-0.0625\epsilon$. Furthermore, if the inequality
$S(\rho||\tau_{\rho})\leq S(\tau_{\rho})-S(\rho)$ is satisfied, from the
identity of Eq. (1), we get the lower bound $\beta\delta_{E}(\rho)\geq
E_{re}(\rho)-S(P_{\rho}||\rho_{\beta})$.

Finally, by considering the measurement induced disturbance, we observe that
$E(\chi^{\prime}_{\rho})=E(\rho)$, since
$\pi_{\chi^{\prime}_{\rho}}=\pi_{\rho}$. We define
$\delta^{\prime}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\chi^{\prime}_{\rho})$,
which is non-negative since $\chi^{\prime}_{\rho}$ is obtained by applying a
unital map to $\rho$, and $\delta^{\prime}(\chi)=0$ if $\chi$ is classical. By
using the identity of Eq. (1), we get

$\beta\delta^{\prime}(\rho)=D^{\prime}(\rho)-S(P_{\rho}||\rho_{\beta})+S(P_{\chi^{\prime}_{\rho}}||\rho_{\beta})$
(12)

and the bounds

$D^{\prime}(\rho)-S(P_{\rho}||\rho_{\beta})\leq\beta\delta^{\prime}(\rho)\leq
D^{\prime}(\rho)+S(P_{\chi^{\prime}_{\rho}}||\rho_{\beta})\,.$ (13)

Of course, similarly to $\delta(\rho)$ we have that if the energy spectrum is
non-degenerate, then $\delta^{\prime}(\rho)=0$ implies that $\rho$ is
classical. In the end, we note that
$\delta_{T}(\rho)=\delta^{\prime}(\rho)+\delta_{C}(\chi^{\prime}_{\rho})$,
thus $\delta_{T}(\rho)$ is negative if
$\delta_{C}(\chi^{\prime}_{\rho})<-\delta^{\prime}(\rho)$. Then,
$\delta_{T}(\rho)$ is non-negative if $\rho$ is a pure state, in contrast for
initial mixed states, similarly to the classical correlations contribution,
for $d_{A}=d_{B}=2$ we have $\delta_{T}(\rho)\geq 0$ and the states with a
negative $\delta_{T}(\rho)$ are unlikely for large $d_{A}$ or $d_{B}$.

For the case of interacting parties, generalization of the correlations
contributions introduced is problematic. It is enough to consider the set of
the product states with the same energy of the initial state. This set can be
empty. To show it, we can consider $\rho=|E_{max}\rangle\langle E_{max}|$,
where $|E_{max}\rangle$ is the state with maximum energy which we assume non-
degenerate. Then, if the two parties interact with each other,
$|E_{max}\rangle$ is entangled and there is not a product state having the
same energy of $\rho$. Thus, for interacting parties, we cannot define the
total correlations contribution by considering the closest product state with
same energy. Alternatively, we can consider the closest product state
$\pi_{\rho}$ and its energy difference with respect to the initial state to
define the total correlations contribution as
$\tilde{\delta}_{T}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\pi_{\rho})-E(\rho)+E(\pi_{\rho})$.
Similarly, we define the quantum correlations contribution as
$\tilde{\delta}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\chi_{\rho})-E(\rho)+E(\chi_{\rho})$
and the entanglement contribution as
$\tilde{\delta}_{E}(\rho)=\mathcal{E}(\rho)-\mathcal{E}(\sigma_{\rho})-E(\rho)+E(\sigma_{\rho})$.
We observe that our main finding, that only the quantum correlations always
give a non-negative contribution to the ergotropy, while the total, classical
and entanglement correlations can give a negative contribution, also applies
to these alternative definitions. We focus on non-interacting parties, we get
$\tilde{\delta}_{T}(\rho)=\delta_{T}(\rho)$ which can be negative, and about
the entanglement contribution, for the example given above,
$\tilde{\delta}_{E}(\rho)=\delta_{E}(\rho)<0$. About the quantum correlations
contribution, we get $\tilde{\delta}(\rho)=E(P_{\chi_{\rho}})-E(P_{\rho})$ and
we have $\tilde{\delta}(\rho)\geq 0$ as for $\delta(\rho)$, since
$\chi_{\rho}$ is obtained by applying a unital map to $\rho$.

Conclusions. In summary, we have identified and investigated the contributions
to the ergotropy coming from different kinds of correlations by comparing the
ergotropy with the one of appropriate closest states. By doing so, we have
established an important link between the field of quantum thermodynamics and
the one of quantum information theory. Relative entropy based measure of
correlations are known very well. By establishing ergotropy based quantifiers
of correlations, we show how ergotropy also plays some major role to
characterize quantum correlations which has less explored so far. In
particular, beyond the foundations implications, the importance of the paper
is also due to the fact that the clarification of the relevance of quantum
features is of central interest in order to achieve quantum thermodynamic
processes with performances classically inhibited. Interestingly, we find that
only the contribution of discord correlations is always positive. Conversely,
other kinds of correlations can reduce the work extraction. We note that the
so-called free energy of the correlated state $\rho$ is not smaller than the
one of the product state $\pi_{\rho}$ of the reduced states (see, e.g., Ref.
bera19 ). Concerning quantum correlations, the free energy of $\rho$ is not
smaller than the one of $\eta_{\rho}$, since $S(\eta_{\rho})\geq S(\rho)$, and
the one of $\tau_{\rho}$, if $S(\tau_{\rho})\geq S(\rho)$, thus the
correlations contributions in terms of the free energy are expected non-
negative. We observe that things can also be different if we consider the work
extraction when the system is in contact with a reservoir. If the reservoir
initially is in a thermal state and uncorrelated with the system, the maximum
work extractable, i.e., the ergotropy of the total system (system plus
reservoir), is upper bounded by minus the change of out-of-equilibrium free
energy of the system, and similar quantifiers can be defined in terms of this
quantity. In particular, it is easy to show that the contribution of total
correlations is always non-negative for non-interacting parties 222To prove
it, we consider the difference of out-of-equilibrium free energy $\Delta
F_{\beta}(\rho)=S(\rho||\rho_{\beta})/\beta$, and by noting that, for non-
interacting parties,
$S(\rho||\rho_{\beta})=-S(\rho)-\text{Tr}\left\\{\rho\ln\rho_{\beta}\right\\}\geq-S(\pi_{\rho})-\text{Tr}\left\\{\pi_{\rho}\ln\rho_{\beta}\right\\}=S(\pi_{\rho}||\rho_{\beta})$,
we get $\Delta F_{\beta}(\rho)\geq\Delta F_{\beta}(\pi_{\rho})$.. In
conclusion, our work provides an essential step forward for the field of
quantum thermodynamics and we hope that it will inspire further investigations
and applications of correlations in this field.

## References

  * (1) A. Einstein, B. Podolsky, and N. Rosen, Phys. Rev. 47, 777 (1935)
  * (2) E. Schrödinger, Naturwissenschaften 23, 844 (1935)
  * (3) N. Bohr, Phys. Rev. 48, 696 (1935)
  * (4) R. Horodecki, P. Horodecki, M. Horodecki and K. Horodecki, Rev. Mod. Phys. 81, 865 (2009)
  * (5) L. Henderson and V. Vedral, J. Phys. A 34, 6899 (2001)
  * (6) H. Ollivier and W. H. Zurek, Phys. Rev. Lett. 88, 017901 (2001)
  * (7) F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso (Eds.), Thermodynamics in the Quantum Regime (Springer,Cham, 2019)
  * (8) A. E. Allahverdyan, R. Balian and Th.M. Nieuwenhuizen, Europhys. Lett. 67, 4 (2004)
  * (9) J. Oppenheim, M. Horodecki, P. Horodecki and R. Horodecki, Phys. Rev. Lett. 89, 180402 (2002)
  * (10) M. Perarnau-Llobet, K. V. Hovhannisyan, M. Huber, P. Skrzypczyk, N. Brunner and A. Acín, Phys. Rev. X 5, 041011 (2015)
  * (11) A. Mukherjee, A. Roy, S. S. Bhattacharya and M. Banik, Phys. Rev. E 93, 052140 (2016)
  * (12) M. Alimuddin, T. Guha and P. Parashar, Phys. Rev. A 99, 052320 (2019)
  * (13) M. Alimuddin, T. Guha and P. Parashar, Phys. Rev. E 102, 012145 (2020)
  * (14) G. Francica, J. Goold, F. Plastina and M. Paternostro, NPJ Quantum Information, 3, 12 (2017)
  * (15) G. Francica, F. C. Binder, G. Guarnieri, M. T. Mitchison, J. Goold and F. Plastina, Phys. Rev. Lett. 125, 180603 (2020)
  * (16) G. Francica, Phys. Rev. E 105, 014101 (2022)
  * (17) K. Modi, T. Paterek, W. Son, V. Vedral and M. Williamson, Phys. Rev. Lett. 104, 080501 (2010)
  * (18) M. A. Nielsen and G. Vidal, Quant. Inf. and Comp. 1, 76 (2001)
  * (19) V. Vedral, M. B. Plenio, K. Jacobs and P. L. Knight, Phys. Rev. A, 56, 4452 (1997)
  * (20) A. Touil, B. Çakmak and S. Deffner, J. Phys. A: Math. Theor. 55, 025301 (2022)
  * (21) V. Vedral and M. B. Plenio, Phys. Rev. A 57, 1619 (1998)
  * (22) M. N. Bera, A. Riera, M. Lewenstein, Z. B. Khanian and A. Winter, Quantum 3, 121 (2019)