# Which Mutual-Information Representation Learning Objectives are Sufficient
for Control?

Kate Rakelly Abhishek Gupta Carlos Florensa Sergey Levine
University of California, Berkeley
{rakelly, abhigupta, florensacc<EMAIL_ADDRESS>Correspondence to
<EMAIL_ADDRESS>
###### Abstract

Mutual information maximization provides an appealing formalism for learning
representations of data. In the context of reinforcement learning (RL), such
representations can accelerate learning by discarding irrelevant and redundant
information, while retaining the information necessary for control. Much of
the prior work on these methods has addressed the practical difficulties of
estimating mutual information from samples of high-dimensional observations,
while comparatively less is understood about _which_ mutual information
objectives yield representations that are sufficient for RL from a theoretical
perspective. In this paper, we formalize the sufficiency of a state
representation for learning and representing the optimal policy, and study
several popular mutual-information based objectives through this lens.
Surprisingly, we find that two of these objectives can yield insufficient
representations given mild and common assumptions on the structure of the MDP.
We corroborate our theoretical results with empirical experiments on a
simulated game environment with visual observations.

##  1 Introduction

Deep reinforcement learning (RL) algorithms are in principle capable of
learning policies from high-dimensional observations, such as camera images
[48, 39, 32]. However, policy learning in practice faces a bottleneck in
acquiring useful representations of the observation space [57]. State
representation learning approaches aim to remedy this issue by learning
structured and compact representations on which to perform RL. A useful state
representation should be _sufficient_ to learn and represent the optimal
policy or the optimal value function, while discarding irrelevant and
redundant information. Understanding whether or not an objective is guaranteed
to yield sufficient representations is important, because insufficient
representations make it impossible to solve certain problems. For example, an
autonomous vehicle would not be able to navigate safely if its state
representation did not contain information about the color of the stoplight in
front of it. With the increasing interest in leveraging offline datasets to
learn representations for RL [19, 35, 63], the question of sufficiency becomes
even more important to understand if the representation is capable of
representing policies and value functions for downstream tasks.

While a wide range of representation learning objectives have been proposed in
the literature [41], in this paper we focus on analyzing sufficiency of
representations learned by maximizing the mutual information (MI) between
random variables. Prior work has proposed many different MI objectives
involving the variables of states, actions, and rewards at different time-
steps [4, 51, 52, 57]. While much prior work has focused on how to optimize
these various mutual information objectives in high dimensions [60, 8, 51,
27], we focus instead on their ability to yield theoretically sufficient
representations. We find that two commonly used objectives are insufficient
for the general class of MDPs, in the most general case, and prove that
another typical objective is sufficient. We illustrate the analysis with both
didactic examples in which MI can be computed exactly and deep RL experiments
in which we approximately maximize the MI objective to learn representations
of visual inputs. The experimental results corroborate our theoretical
findings, and demonstrate that the sufficiency of a representation can have a
substantial impact on the performance of an RL agent that uses that
representation. This paper provides guidance to the deep RL practitioner on
when and why objectives may be expected to work well or fail, and also
provides a formal framework to analyze newly proposed representation learning
objectives based on MI.

##  2 Related Work

In this paper, we analyze several widely used mutual information objectives
for control. In this section we first review MI-based unsupervised learning,
then the application of these techniques to the RL setting. Finally, we
discuss alternative perspectives on representation learning in RL.

Mutual information-based unsupervised learning. A common technique for
unsupervised representation learning based on the InfoMax principle [43, 9] is
to maximize the MI between the input and its latent representation subject to
domain-specific constraints [7]. This technique has been applied to learn
representations for natural language [14], video [64], and images [5, 27] and
even policy learning via RL in high dimensions [61]. To address the
difficulties of estimating MI from samples [45] and with high-dimensional
inputs [60], much recent work has focused on improving MI estimation via
variational methods [60, 54, 51, 8]. In this work we are concerned with
analyzing the MI objectives, and not the estimation method. In our experiments
with image observations, we use noise contrastive estimation methods [24],
though other choices could also suffice.

Mutual information objectives in RL. Reinforcement learning adds aspects of
temporal structure and control to the standard unsupervised learning problem
discussed above (see Figure 1). This structure can be leveraged by maximizing
MI between sequential states, actions, or combinations thereof. Some works
omit the action, maximizing the MI between current and future states [4, 51,
63]. Several prior works [50, 56, 59, 44] maximize MI objectives that closely
resemble the forward information objective we introduce in Section 4, whiel
others optimize related objectives by learning latent forward dynamics models
[69, 33, 73, 26, 39]. Multi-step inverse models, closely related to the
inverse information objective (Section 4), have been used to learn control-
centric representations [70, 23]. Single-step inverse models have been
deployed as regularization of forward models [72, 2] and as an auxiliary loss
for policy gradient RL [57, 52]. The MI objectives that we study have also
been used as reward bonuses to improve exploration, without impacting the
representation, in the form of empowerment [37, 36, 49, 40] and information-
theoretic curiosity [62].

Representation learning for reinforcement learning. In RL, the problem of
finding a compact state space has been studied as state aggregation or
abstraction [6, 42]. Abstraction schemes include bisimulation [22],
homomorphism [55], utile distinction [46], and policy irrelevance [30]. While
efficient algorithms exist for MDPs with known transition models for some
abstraction schemes such as bisimulation [18, 22], in general obtaining error-
free abstractions is impractical for most problems of interest. For
approximate abstractions prior work has bounded the sub-optimality of the
policy [11, 13, 1] as well as the sample efficiency [38, 67, 16], with some
results in the deep learning setting [21, 50]. In this paper, we focus on
whether a representation can be used to learn the optimal policy, and not the
tractability of learning. Li et al. [42] shares this focus; while they
establish convergence properties of $Q$-learning with representations
satisfying different notions of sufficiency, we leverage their
$Q^{*}$-sufficiency criteria to evaluate representations learned via MI-based
objectives. Alternative approaches to representation learning for RL include
priors based on the structure of the physical world [31] or heuristics such as
disentanglement [66], meta-learning general value functions [68], predicting
multiple value functions [10, 17, 29] and predicting domain-specific
measurements [47, 15]. We restrict our analysis to objectives that can be
expressed as MI-maximization. In our paper we focus on the representation
learning problem, disentangled from exploration, a strategy shared by prior
works [19, 35, 63, 71].

##  3 Representation Learning for RL

The goal of representation learning for RL is to learn a compact
representation of the state space that discards irrelevant and redundant
information, while still retaining sufficient information to represent
policies and value functions needed for learning. In this section we formalize
this problem, and propose and define the concept of sufficiency to evaluate
the usefulness of a representation.

###  3.1 Preliminaries

We begin with brief preliminaries of reinforcement learning and mutual
information.

##### Reinforcement learning.

A Markov decision process (MDP) is defined by the tuple
$(\mathcal{S},\mathcal{A},\mathcal{T},r)$, where $\mathcal{S}$ is the set of
states, $\mathcal{A}$ the set of actions,
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$
the state transition distribution, and $r:\mathcal{S}\rightarrow\mathbb{R}$
the reward function 111We restrict our attention to MDPs where the reward can
be expressed as a function of the state, which is fairly standard across a
broad set of real world RL problems. We will use capital letters to refer to
random variables and lower case letters to refer to values of those variables
(e.g., $S$ is the random variable for the state and $\mathbf{s}$ is a specific
state). Throughout our analysis we will often be interested in multiple reward
functions, and denote a set of reward functions as $\mathcal{R}$. The
objective of RL is to find a policy that maximizes the sum of discounted
returns $\bar{R}$ for a given reward function $r$, and we denote this optimal
policy as
$\pi^{*}_{r}=\operatorname*{arg\,max}_{\pi}\mathbb{E}_{\pi}[\sum_{t}\gamma^{t}r(S_{t},A_{t})]$
for discount factor $\gamma$. We also define the optimal $Q$-function as
$Q^{*}_{r}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbb{E}_{\pi^{*}}[\sum_{t=1}^{\infty}\gamma^{t}r(S_{t},A_{t})|\mathbf{s}_{t},\mathbf{a}_{t}]$.
The optimal $Q$-function satisfies the recursive Bellman equation,
$Q^{*}_{r}(\mathbf{s}_{t},\mathbf{a}_{t})=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\mathbb{E}_{p(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})}\operatorname*{arg\,max}_{\mathbf{a}_{t+1}}Q^{*}_{r}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})$.
An optimal policy and the optimal Q-function are related according to
$\pi^{*}(\mathbf{s})=\operatorname*{arg\,max}_{\mathbf{a}}Q^{*}(\mathbf{s},\mathbf{a})$.

##### Mutual information.

In information theory, the mutual information (MI) between two random
variables, $X$ and $Y$, is defined as [12]:

$I(X;Y)=\mathbb{E}_{p(x,y)}\log\frac{p(x,y)}{p(x)p(y)}=H(X)-H(X|Y).$ (1)

The first definition indicates that MI can be understood as a relative entropy
(or KL-divergence), while the second underscores the intuitive notion that MI
measures the reduction in the uncertainty of one random variable from
observing the value of the other.

##### Representation learning for RL.

Figure 1:  Probabilistic graphical model illustrating the state representation
learning problem: estimate representation $Z$ from original state $S$.

The goal of representation learning for RL is to find a compact representation
of the state space that discards details in the state that are not relevant
for representing the policy or value function, while preserving task-relevant
information (see Figure 1). While state aggregation methods typically define
deterministic rules to group states in the representation [6, 42], MI-based
representation learning methods used for deep RL treat the representation as a
random variable [50, 51, 52]. Accordingly, we formalize a representation as a
stochastic mapping between original state space and representation space.

######  Definition 1.

A stochastic representation $\phi_{\mathcal{Z}}(\mathbf{s})$ is a mapping from
states $\mathbf{s}\in\mathcal{S}$ to a probability distribution
$p(Z|S=\mathbf{s})$ over elements of a new representation space
$z\in\mathcal{Z}$.

In this work we consider learning state representations from data by
maximizing an objective $\mathbb{J}$. Given an objective $\mathbb{J}$, we
define the set of representations that maximize this objective as
$\Phi_{\mathbb{J}}=\\{\phi_{\mathcal{Z}}~{}|~{}\phi_{\mathcal{Z}}\in\operatorname*{arg\,max}\mathbb{J}(\phi)$.
Unlike problem formulations for partially observed settings [69, 26, 39], we
assume that $S$ is a Markovian state; therefore the representation for a given
state is conditionally independent of the past states, a common assumption in
the state aggregation literature [6, 42]. See Figure 1 for a depiction of the
graphical model.

###  3.2 Sufficient Representations for Reinforcement Learning

We now turn to the problem of evaluating stochastic representations for RL.
Intuitively, we expect a useful state representation to be capable of
representing an optimal policy in the original state space.

######  Definition 2.

A representation $\phi_{\mathcal{Z}}$ is $\pi^{*}$-sufficient with respect to
a set of reward functions $\mathcal{R}$ if $\forall r\in\mathcal{R}$,
$\phi_{\mathcal{Z}}(\mathbf{s}_{1})=\phi_{\mathcal{Z}}(\mathbf{s}_{2})\implies\pi^{*}_{r}(A|\mathbf{s}_{1})=\pi^{*}_{r}(A|\mathbf{s}_{2})$.

When a stochastic representation $\phi_{\mathcal{Z}}$ produces the same
distribution over the representation space for two different states
$\mathbf{s}_{1}$ and $\mathbf{s}_{2}$ we say it aliases these states.
Unfortunately, as already proven in Theorem 4 of Li et al. [42] for the more
restrictive case of deterministic representations, being able to represent the
optimal policy does not guarantee that it can be learned via RL in the
representation space. Accordingly, we define a stricter notion of sufficiency
that _does_ guarantee the convergence of Q-learning to the optimal policy in
the original state space (refer to Theorem 4 of Li et al. [42] for the proof
of this).

######  Definition 3.

A representation $\phi_{\mathcal{Z}}$ is $Q^{*}$-sufficient with respect to a
set of reward functions $\mathcal{R}$ if $\forall r\in\mathcal{R}$,
$\phi_{\mathcal{Z}}(\mathbf{s}_{1})=\phi_{\mathcal{Z}}(\mathbf{s}_{2})\implies\forall\mathbf{a}\text{,}Q^{*}_{r}(\mathbf{a},\mathbf{s}_{1})=Q^{*}_{r}(\mathbf{a},\mathbf{s}_{2})$.

Note that $Q^{*}$-sufficiency implies $\pi^{*}$-sufficiency since an optimal
policy can be recovered from the optimal Q-function via
$\pi^{*}_{r}(s)=\operatorname*{arg\,max}_{a}Q^{*}_{r}(s,a)$ [65]; however the
converse is not true. We emphasize that while $Q^{*}$-sufficiency guarantees
convergence, it does not guarantee tractability, which has been explored in
prior work [38, 16].

We will further say that an _objective_ $\mathbb{J}$ is sufficient with
respect to some set of reward functions $\mathcal{R}$ if all the
representations that maximize that objective $\Phi_{\mathbb{J}}$ are
sufficient with respect to every element of $\mathcal{R}$ according to the
definition above. Surprisingly, we will demonstrate that not all commonly used
objectives satisfy this basic qualification even when $\mathcal{R}$ contains a
single known reward function.

##  4 Mutual Information for Representation Learning in RL

In our study, we consider several MI objectives proposed in the literature.

##### Forward information:

A commonly sought characteristic of a state representation is to ensure it
retains maximum predictive power over future state representations. This
property is satisfied by representations maximizing the following MI
objective,

$\mathbb{J}_{fwd}=I(Z_{t+1};Z_{t},A_{t})=H(Z_{t+1})-H(Z_{t+1}|Z_{t},A_{t}).$
(2)

We suggestively name this objective “forward information” due to the second
term, which is the entropy of the forward dynamics distribution. This
objective and closely related ones have been used in prior works [50, 56, 59,
44].

##### State-only transition information:

Several popular methods [51, 4, 63] optimize a similar objective, but do not
include the action:

$\mathbb{J}_{state}=I(Z_{t+k};Z_{t})=H(Z_{t+k})-H(Z_{t+k}|Z_{t}).$ (3)

As we will show, the exclusion of the action can have a profound effect on the
characteristics of the resulting representations.

##### Inverse information:

Another commonly sought characteristic of state representations is to retain
maximum predictive power of the action distribution that could have generated
an observed transition from $\mathbf{s}_{t}$ to $\mathbf{s}_{t+1}$. Such
representations can be learned by maximizing the following information
theoretic objective:

$\mathbb{J}_{inv}=I(A_{t};Z_{t+k}|Z_{t})=H(A_{t}|Z_{t})-H(A_{t}|Z_{t},Z_{t+k})$
(4)

We suggestively name this objective “inverse information” due to the second
term, which is the entropy of the inverse dynamics. A wide range of prior work
learns representations by optimizing closely related objectives [23, 57, 2,
52, 70, 72]. Intuitively, inverse models allow the representation to capture
only the elements of the state that are necessary to predict the action,
allowing the discard of potentially irrelevant information.

##  5 Sufficiency Analysis

In this section we analyze the sufficiency for control of representations
obtained by maximizing each objective presented in Section 4. To focus on the
representation learning problem, we decouple it from RL by assuming access to
a dataset of transitions collected with a policy that reaches all states with
non-zero probability, which can then be used to learn the desired
representation. We also assume that distributions, such as the dynamics or
inverse dynamics, can be modeled with arbitrary accuracy, and that the
maximizing set of representations for a given objective can be computed. While
these assumptions might be relaxed in any practical RL algorithm, and
exploration plays a confounding role, the ideal assumptions underlying our
analysis provide the best-case scenario for objectives to yield provably
sufficient representations. In other words, objectives found to be provably
insufficient under ideal conditions will continue to be insufficient under
more realistic assumptions.

###  5.1 Forward Information

In this section we show that a representation that maximizes
$\mathbb{J}_{fwd}$ is sufficient for optimal control under any reward
function. This result aligns with the intuition that a representation that
captures forward dynamics can represent everything predictable in the state
space, and can thus be used to learn the optimal policy for any task. This
strength can also be a weakness if there are many predictable elements that
are irrelevant for downstream tasks, since the representation retains more
information than is needed for the task. Note that the representation can
still discard information in the original state, such as independent random
noise at each timestep.

######  Proposition 1.

$\mathbb{J}_{fwd}$ is sufficient for all reward functions.

###### Proof.

_(Sketch)_ We first show in Lemma 1 that if $Z_{t},A_{t}$ are maximally
informative of $Z_{t+k}$, they are also maximally informative of the return
$\bar{R}_{t}$. Thanks to the Markov structure, we then show in Lemma 2 that
$\mathbb{E}_{p(Z_{t}|S_{t}=\mathbf{s})}p(\bar{R}_{t}|Z_{t},A_{t})=p(\bar{R}_{t}|S_{t}=\mathbf{s},A_{t})$.
In other words, given $\phi_{\mathcal{Z}}$, additionally knowing $S$ doesn’t
change our belief about the future return. The $Q$-value is the expectation of
the return, so $Z$ has as much information about the $Q$-value as $S$. See
Appendix A.1 for the proof. ∎

###  5.2 State-Only Transition Information

Figure 2: (left) A representation that aliases the states $\mathbf{s}_{0}$ and
$\mathbf{s}_{3}$ into a single state maximizes $\mathbb{J}_{state}$ but is not
sufficient to represent the optimal policy which must choose different actions
in $\mathbf{s}_{0}$ and $\mathbf{s}_{3}$ to reach $\mathbf{s}_{2}$ which
yields reward. (right) Values of $\mathbb{J}_{state}$ and $\mathbb{J}_{fwd}$
for a few representative state representations, ordered by increasing
$I(Z;S)$. The representation that aliases $\mathbf{s}_{0}$ and
$\mathbf{s}_{3}$ (plotted with a diamond) maximizes $\mathbb{J}_{state}$, but
the policy learned with this representation may not be optimal (as shown
here). The original state representation (plotted with a star) is sufficient.

While $\mathbb{J}_{state}$ is closely related to $\mathbb{J}_{fwd}$, we now
show that $\mathbb{J}_{state}$ is not sufficient.

######  Proposition 2.

$\mathbb{J}_{state}$ is not sufficient for all reward functions.

###### Proof.

We show this by counter-example with the deterministic-transition MDP defined
in Figure 2 (left). For all four states, let the two actions $\mathbf{a}_{0}$
and $\mathbf{a}_{1}$ be equally likely under the policy distribution. In this
case, each state gives no information about which of the two possible next
states is more likely; this depends on the action. Therefore, a representation
maximizing $\mathbb{J}_{state}$ is free to alias states with the same next-
state distribution, such as $\mathbf{s}_{0}$ and $\mathbf{s}_{3}$. An
alternative view is that such a representation can maximize
$\mathbb{J}_{state}=H(Z_{t+k})-H(Z_{t+k}|Z_{t})$ by reducing both terms in
equal amounts - aliasing $\mathbf{s}_{0}$ and $\mathbf{s}_{3}$ decreases the
marginal entropy as well as the entropy of predicting the next state starting
from $\mathbf{s}_{1}$ or $\mathbf{s}_{2}$. However, this aliased
representation is not capable of representing the optimal policy, which must
distinguish $\mathbf{s}_{0}$ and $\mathbf{s}_{3}$ in order to choose the
correct action to reach $\mathbf{s}_{2}$, which yields reward. ∎

In Figure 2 (right), we illustrate the insufficiency of $\mathbb{J}_{state}$
computationally, by computing the values of $\mathbb{J}_{state}$ and
$\mathbb{J}_{fwd}$ for different state representations of the above MDP,
ordered by increasing $I(Z;S)$. At the far left of the plot is the
representation that aliases all states, while the original state
representation is at the far right (plotted with a star). The representation
that aliases states $\mathbf{s}_{0}$ and $\mathbf{s}_{3}$ (plotted with a
diamond) maximizes $\mathbb{J}_{state}$, but is insufficient to represent the
optimal policy. Value iteration run with this state representation achieves
only half the optimal return ($0.5$ versus $1.0$).

###  5.3 Inverse Information

Figure 3: (left) In this MDP, a representation that aliases the states
$\mathbf{s}_{0}$ and $\mathbf{s}_{1}$ into a single state maximizes
$\mathbb{J}_{inv}$, yet is not sufficient to represent the optimal policy,
which must distinguish between $\mathbf{s}_{0}$ and $\mathbf{s}_{1}$ in order
to take a different action (towards the high-reward states outlined in green).
(right) Values of $\mathbb{J}_{inv}$ and $\mathbb{J}_{fwd}$ for a few selected
state representations, ordered by increasing $I(Z;S)$. The representation that
aliases $\mathbf{s}_{0}$ and $\mathbf{s}_{1}$ (plotted with a diamond)
maximizes $\mathbb{J}_{inv}$, but is not sufficient to learn the optimal
policy. Note that this counterexample holds also for
$\mathbb{J}_{inv}+I(R;Z)$.

Here we show that representations that maximize $\mathbb{J}_{inv}$ are not
sufficient for control in all MDPs. Intuitively, one way that the
representation can be insufficient is by retaining only controllable state
elements, while the reward function depends on state elements outside the
agent’s control. We then show that additionally representing the immediate
reward is not enough to resolve this issue.

######  Proposition 3.

$\mathbb{J}_{inv}$ is not sufficient for all reward functions. Additionally,
adding $I(R_{t};Z_{t})$ to the objective does not make it sufficient.

###### Proof.

We show this by counter-example with the deterministic-transition MDP defined
in Figure 3 (left). Now consider the representation that aliases the states
$\mathbf{s}_{0}$ and $\mathbf{s}_{1}$. This state representation wouldn’t be
sufficient for control because the same actions taken from these two states
lead to different next states, which have different rewards ($\mathbf{a}_{0}$
leads to the reward from $\mathbf{s}_{0}$ while $\mathbf{a}_{1}$ leads to the
reward from $\mathbf{s}_{1}$). However, this representation maximizes
$\mathbb{J}_{inv}$ because, given each pair of states, the action is
identifiable. Interestingly, this problem cannot be remedied by simply
requiring that the representation also be capable of predicting the reward at
each state. Indeed, the same insufficient representation from the above
counterexample also maximizes this new objective as long as the reward at
$\mathbf{s}_{0}$ and $\mathbf{s}_{1}$ are the same. ∎

Analogous to the preceding section, in Figure 3 (right), we plot the values of
the objectives $\mathbb{J}_{inv}$ and $\mathbb{J}_{fwd}$ for state
representations ordered by increasing $I(Z;S)$. The representation that
aliases states $\mathbf{s}_{0}$ and $\mathbf{s}_{1}$ (plotted with a diamond)
maximizes $\mathbb{J}_{inv}$, but is insufficient to represent the optimal
policy; value iteration with this state representation achieves half the
optimal return ($0.5$ versus $1.0$).

##  6 Experiments

To analyze whether the conclusions of our theoretical analysis hold in
practice, we present experiments studying MI-based representation learning
with image observations. We do not aim to show that any particular method is
necessarily better or worse, but rather to determine whether the sufficiency
arguments that we presented can translate into quantifiable performance
differences in deep RL.

###  6.1 Experimental Setup

To separate representation learning from RL, we first optimize each
representation learning objective on a dataset of offline data, similar to the
protocol in Stooke et al. [63]. Our datasets consist of $50$k transitions
collected from a uniform random policy, which is sufficient to cover the state
space in our environments. We then freeze the weights of the state encoder
learned in the first phase and train RL agents with the representation as
state input. We perform our experiments on variations on the pygame [58] video
game catcher, in which the agent controls a paddle that it can move back and
forth to catch fruit that falls from the top of the screen (see Figure 4). A
positive reward is given when the fruit is caught and a negative reward when
the fruit is not caught. The episode terminates after one piece of fruit
falls. We optimize $\mathbb{J}_{fwd}$ and $\mathbb{J}_{state}$ with noise
contrastive estimation [24], and $\mathbb{J}_{inv}$ by training an inverse
model via maximum likelihood. For the RL algorithm, we use the Soft Actor-
Critic algorithm [25], modified slightly for the discrete action distribution.
Please see Appendix A.2 for full experimental details.

Figure 4: (left) Original catcher game in which the agent (grey paddle) moves
left or right to catch fruit (yellow square) that falls from the top of the
screen. (right) Variation catcher-grip in which the agent is instantiated as a
gripper, and must open the gripper to catch fruit.

###  6.2 Computational Results

In principle, we expect that a representation learned with $\mathbb{J}_{inv}$
may not be sufficient to solve the catcher game. Because the agent does not
control the fruit, a representation maximizing $\mathbb{J}_{inv}$ might
discard that information, thereby making it impossible to represent the
optimal policy. We observe in Figure 5 (top left) that indeed the
representation trained to maximize $\mathbb{J}_{inv}$ results in RL agents
that converge slower and to a lower asymptotic expected return. Further,
attempting to learn a decoder from the learned representation to the position
of the falling fruit incurs a high error (Figure 5, bottom left), indicating
that the fruit is not precisely captured by the representation. The
characteristics of this simulated game are representative of realistic tasks.
Consider, for instance, an autonomous vehicle that is stopped at a stoplight.
Because the agent does not control the color of the stoplight, it may not be
captured in the representation learned by $\mathbb{J}_{inv}$ and the resulting
RL policy may choose to run the light.

Obj. | Agent Err. | Fruit Err.
---|---|---
$\mathbb{J}_{fwd}$ | 0.1 | 0.1
$\mathbb{J}_{inv}$ | 0.15 | 0.37

Obj. | Agent Err. | Fruit Err. | Grip Err.
---|---|---|---
$\mathbb{J}_{fwd}$ | 0.1 | 0.1 | 0.3
$\mathbb{J}_{state}$ | 0.1 | 0.1 | 0.46

Figure 5:  (left) Policy performance using learned representations as state
inputs to RL, for the catcher and catcher-grip environments. (right) Error in
predicting the positions of ground truth state elements from each learned
representation. Representations maximizing $\mathbb{J}_{inv}$ need not
represent the fruit, while representations maximizing $\mathbb{J}_{state}$
need not represent the gripper, leading these representations to perform
poorly in catcher and catcher-grip respectively.

In the second experiment, we consider a failure mode of $\mathbb{J}_{state}$.
We augment the paddle with a gripper that the agent controls and must be open
in order to properly catch the fruit (see Figure 4, right). Since the change
in the gripper is completely controlled by a single action, the current state
contains no information about the state of the gripper in the future.
Therefore, a representation maximizing $\mathbb{J}_{state}$ might alias states
where the gripper is open with states where the gripper is closed. Indeed, we
see that the error in predicting the state of the gripper from the
representation learned via $\mathbb{J}_{state}$ is about chance (Figure 5,
bottom right). This degrades the performance of an RL agent trained with this
state representation since the best the agent can do is move under the fruit
and randomly open or close the gripper (Figure 5, top right). In the driving
example, suppose turning on the headlights incurs positive reward if it’s
raining but negative reward if it’s sunny. The representation could fail to
distinguish the state of the headlights, making it impossible to learn when to
properly use them. $\mathbb{J}_{fwd}$ produces useful representations in all
cases, and is equally or more effective than learning representations purely
from the RL objective alone (Figure 5).

###  6.3 Increasing visual complexity via background distractors

In this section, we test whether sufficiency of the state representation
impacts agent performance in more visually complex environments by adding
background distractors to the agent’s observations. We randomly generate
images of $10$ circles of different colors and replace the background of the
game with these images. See Figure 6 for examples of the agent’s observations.

Figure 6: Example $64$x$64$ pixel agent observations with background
distractors (colored circles) randomly generated at each time step. The
distractors increase the difficulty of learning a good representation, which
must capture both agent and fruit.

Analogous to Section 6, in Figure 7 we show the performance of an RL agent
trained with the frozen representation as input (top), as well as the error of
decoding true state elements from the representation (bottom). In both games,
end-to-end RL from images performs poorly, demonstrating the need for
representation learning to aid in solving the task. As predicted by the
theory, the representation learned by $\mathbb{J}_{inv}$ fails in both games,
and the representation learned by $\mathbb{J}_{state}$ fails in the catcher-
grip game. The difference in performance between sufficient and insufficient
objectives is even more pronounced in this setting than in the plain
background setting. With more information present in the observation in the
form of the distractors, insufficient objectives that do not optimize for
representing all the required state information may be “distracted” by
representing the background objects instead, resulting in low performance. In
Appendix A.5 we experiment with visual distractors that are temporally
correlated across time. We also consider variations on our analysis,
evaluating how well the representations predict the predict the optimal
$Q^{*}$ in Appendix A.3, and experimenting with a different data distribution
for collecting the representation learning dataset in Appendix A.4.

Obj. | Agent Err. | Fruit Err.
---|---|---
$\mathbb{J}_{fwd}$ | 0.13 | 0.14
$\mathbb{J}_{inv}$ | 0.15 | 0.47

Obj. | Agent Err. | Fruit Err. | Grip Err.
---|---|---|---
$\mathbb{J}_{fwd}$ | 0.12 | 0.1 | 0.25
$\mathbb{J}_{state}$ | 0.1 | 0.14 | 0.5

Figure 7: With background distractors added to the observations, the state
representation learned via $\mathbb{J}_{inv}$ fails to capture the fruit
object accurately in catcher (left), and the representation learned via
$\mathbb{J}_{state}$ continue to perform poorly at capturing the gripper state
in catcher-grip (right). The performance of the insufficient representations
is even lower than in the clean background experiment.

##  7 Discussion

In this work, we analyze which common MI-based representation learning
objectives are guaranteed to yield representations provably sufficient for
learning the optimal policy. We show that two common objectives
$\mathbb{J}_{state}$ and $\mathbb{J}_{inv}$ yield theoretically insufficient
representations, and provide a proof of sufficiency for $\mathbb{J}_{fwd}$. We
then show that insufficiency of representations can degrade the performance of
deep RL agents with experiments on a simulated environment with visual
observations. While an insufficient representation learning objective _can_
work well for training RL agents on simulated benchmark environments, the same
objective may fail for a real-world system with different characteristics. We
believe that encouraging a focus on evaluating the sufficiency of newly
proposed representation learning objectives can help better predict potential
failures.

Of the objectives studied in our analysis, only $\mathbb{J}_{fwd}$ is
sufficient to represent optimal Q-functions for any reward function. Note
however that the representation obtained by this objective lacks a notion of
“task-relevance” as it must be equally predictive of all predictable elements
in the state. Therefore an interesting related question is whether it is
possible to construct a general representation that is sufficient for any
policy but contains less information than the smallest representation that
maximizes $\mathbb{J}_{fwd}$. We conjecture that the answer is no, but leave
investigation of this point for future work.

In addition to this direction, we see many other exciting avenues for future
work. First, identifying more restrictive MDP classes in which insufficient
objectives are in fact sufficient. For example, $\mathbb{J}_{state}$ is
trivially sufficient when the environment dynamics and the agent’s policy are
deterministic. However, we hypothesize there may be more interesting MDP
classes, related to realistic applications, in which generally insufficient
objectives may be sufficient. Additionally, extending our analysis to the
partially observed setting would be more reflective of practical applications.
An interesting class of models to consider in this context are generative
models such as variational auto-encoders [34]. Prior work has shown that
maximizing the ELBO alone cannot control the content of the learned
representation [28, 53, 3]. We conjecture that the zero-distortion maximizer
of the ELBO would be sufficient, while other solutions would not necessarily
be. We see these directions as fruitful in providing a deeper understanding of
the learning dynamics of deep RL, and potentially yielding novel algorithms
for provably accelerating RL with representation learning.

Acknowledgements. We would like to thank Ignasi Clavera, Chelsea Finn, and Ben
Poole for insightful conversations at different stages of this work. This
research was supported by the DARPA Assured Autonomy program and ARL DCIST CRA
W911NF-17-2-0181.

## References

  * Abel et al. [2016] D. Abel, D. E. Hershkowitz, and M. L. Littman.  Near optimal behavior via approximate state abstraction.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48_ , pages 2915–2923, 2016\.
  * Agrawal et al. [2016] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine.  Learning to poke by poking: Experiential learning of intuitive physics.  In _Advances in neural information processing systems_ , pages 5074–5082, 2016.
  * Alemi et al. [2018] A. A. Alemi, B. Poole, I. Fischer, J. V. Dillon, R. A. Saurous, and K. Murphy.  Fixing a broken elbo.  In _International Conference on Machine Learning (ICML)_ , 2018.
  * Anand et al. [2019] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm.  Unsupervised state representation learning in atari.  In _Advances in Neural Information Processing Systems_ , pages 8766–8779, 2019.
  * Bachman et al. [2019] P. Bachman, R. D. Hjelm, and W. Buchwalter.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems_ , pages 15509–15519, 2019.
  * Bean et al. [1987] J. C. Bean, J. R. Birge, and R. L. Smith.  Aggregation in dynamic programming.  _Operations Research_ , 35(2):215–220, 1987.
  * Becker and Hinton [1992] S. Becker and G. E. Hinton.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  _Nature_ , 355(6356):161–163, 1992.
  * Belghazi et al. [2018] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm.  Mine: mutual information neural estimation.  _arXiv preprint arXiv:1801.04062_ , 2018.
  * Bell and Sejnowski [1995] A. J. Bell and T. J. Sejnowski.  An information-maximization approach to blind separation and blind deconvolution.  _Neural computation_ , 7(6):1129–1159, 1995.
  * Bellemare et al. [2019] M. Bellemare, W. Dabney, R. Dadashi, A. A. Taiga, P. S. Castro, N. Le Roux, D. Schuurmans, T. Lattimore, and C. Lyle.  A geometric perspective on optimal representations for reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pages 4360–4371, 2019.
  * Bertsekas et al. [1988] D. P. Bertsekas, D. A. Castanon, et al.  Adaptive aggregation methods for infinite horizon dynamic programming.  1988\.
  * Cover [1999] T. M. Cover.  _Elements of information theory_.  John Wiley & Sons, 1999.
  * Dean and Givan [1997] T. Dean and R. Givan.  Model minimization in markov decision processes.  In _AAAI/IAAI_ , pages 106–111, 1997.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, 2019.
  * Dosovitskiy and Koltun [2016] A. Dosovitskiy and V. Koltun.  Learning to act by predicting the future.  In _International Conference on Learning Representations_ , 2016.
  * Du et al. [2019] S. S. Du, S. M. Kakade, R. Wang, and L. F. Yang.  Is a good representation sufficient for sample efficient reinforcement learning?  _arXiv preprint arXiv:1910.03016_ , 2019.
  * Fedus et al. [2019] W. Fedus, C. Gelada, Y. Bengio, M. G. Bellemare, and H. Larochelle.  Hyperbolic discounting and learning over multiple horizons.  _arXiv preprint arXiv:1902.06865_ , 2019.
  * Ferns et al. [2006] N. Ferns, P. S. Castro, D. Precup, and P. Panangaden.  Methods for computing state similarity in markov decision processes.  In _Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence_ , pages 174–181, 2006.
  * Finn et al. [2016] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel.  Deep spatial autoencoders for visuomotor learning.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 512–519. IEEE, 2016.
  * garage contributors [2019] T. garage contributors.  Garage: A toolkit for reproducible reinforcement learning research.  https://github.com/rlworkgroup/garage, 2019.
  * Gelada et al. [2019] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare.  Deepmdp: Learning continuous latent space models for representation learning.  In _International Conference on Machine Learning_ , pages 2170–2179, 2019.
  * Givan et al. [2003] R. Givan, T. Dean, and M. Greig.  Equivalence notions and model minimization in markov decision processes.  _Artificial Intelligence_ , 147(1-2):163–223, 2003.
  * Gregor et al. [2016] K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Gutmann and Hyvärinen [2010] M. Gutmann and A. Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pages 297–304, 2010.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018.
  * Hafner et al. [2019] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International Conference on Machine Learning_ , pages 2555–2565, 2019.
  * Hjelm et al. [2018] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, A. Trischler, and Y. Bengio.  Learning deep representations by mutual information estimation and maximization.  _arXiv preprint arXiv:1808.06670_ , 2018.
  * Huszár [2017] F. Huszár.  Is maximum likelihood useful for representation learning?, 2017.
  * Jaderberg et al. [2016] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  _arXiv preprint arXiv:1611.05397_ , 2016.
  * Jong and Stone [2005] N. K. Jong and P. Stone.  State abstraction discovery from irrelevant state variables.  In _IJCAI_ , volume 8, pages 752–757, 2005.
  * Jonschkowski and Brock [2015] R. Jonschkowski and O. Brock.  Learning state representations with robotic priors.  _Autonomous Robots_ , 39(3):407–428, 2015.
  * Kalashnikov et al. [2018] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning_ , pages 651–673, 2018.
  * Karl et al. [2016] M. Karl, M. Soelch, J. Bayer, and P. van der Smagt.  Deep variational bayes filters: Unsupervised learning of state space models from raw data.  In _International Conference on Learning Representations_ , 2016.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kipf et al. [2019] T. Kipf, E. van der Pol, and M. Welling.  Contrastive learning of structured world models.  _arXiv preprint arXiv:1911.12247_ , 2019.
  * Klyubin et al. [2005] A. S. Klyubin, D. Polani, and C. L. Nehaniv.  Empowerment: A universal agent-centric measure of control.  In _2005 IEEE Congress on Evolutionary Computation_ , volume 1, pages 128–135. IEEE, 2005.
  * Klyubin et al. [2008] A. S. Klyubin, D. Polani, and C. L. Nehaniv.  Keep your options open: An information-based driving principle for sensorimotor systems.  _PloS one_ , 3(12), 2008.
  * Lattimore and Szepesvari [2019] T. Lattimore and C. Szepesvari.  Learning with good feature representations in bandits and in rl with a generative model.  _arXiv preprint arXiv:1911.07676_ , 2019.
  * Lee et al. [2019] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  _arXiv preprint arXiv:1907.00953_ , 2019.
  * Leibfried et al. [2019] F. Leibfried, S. Pascual-Díaz, and J. Grau-Moya.  A unified bellman optimality principle combining reward maximization and empowerment.  In _Advances in Neural Information Processing Systems_ , pages 7867–7878, 2019.
  * Lesort et al. [2018] T. Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and D. Filliat.  State representation learning for control: An overview.  _Neural Networks_ , 108:379–392, 2018.
  * Li et al. [2006] L. Li, T. J. Walsh, and M. L. Littman.  Towards a unified theory of state abstraction for mdps.  In _International Symposium on Artificial Intelligence and Mathematics, ISAIM 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006_ , 2006\.  URL http://anytime.cs.umass.edu/aimath06/proceedings/P21.pdf.
  * Linsker [1988] R. Linsker.  Self-organization in a perceptual network.  _Computer_ , 21(3):105–117, 1988.
  * Mazoure et al. [2020] B. Mazoure, R. T. d. Combes, T. Doan, P. Bachman, and R. D. Hjelm.  Deep reinforcement and infomax learning.  _arXiv preprint arXiv:2006.07217_ , 2020.
  * McAllester and Statos [2018] D. McAllester and K. Statos.  Formal limitations on the measurement of mutual information.  _arXiv preprint arXiv:1811.04251_ , 2018.
  * McCallum [1996] A. K. McCallum.  _Reinforcement Learning with Selective Perception and Hidden State_.  PhD thesis, University of Rochester, 1996.
  * Mirowski [2019] P. Mirowski.  Learning to navigate.  In _1st International Workshop on Multimodal Understanding and Learning for Embodied Applications_ , pages 25–25, 2019.
  * Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mohamed and Rezende [2015] S. Mohamed and D. J. Rezende.  Variational information maximisation for intrinsically motivated reinforcement learning.  In _Advances in neural information processing systems_ , pages 2125–2133, 2015.
  * Nachum et al. [2018] O. Nachum, S. Gu, H. Lee, and S. Levine.  Near-optimal representation learning for hierarchical reinforcement learning.  2018\.
  * Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 2778–2787. JMLR. org, 2017.
  * Phuong et al. [2018] M. Phuong, M. Welling, N. Kushman, R. Tomioka, and S. Nowozin.  The mutual autoencoder: Controlling information in latent code representations.  2018\.
  * Poole et al. [2019] B. Poole, S. Ozair, A. v. d. Oord, A. A. Alemi, and G. Tucker.  On variational bounds of mutual information.  _arXiv preprint arXiv:1905.06922_ , 2019.
  * Ravindran and Barto [2003] B. Ravindran and A. G. Barto.  Smdp homomorphisms: an algebraic approach to abstraction in semi-markov decision processes.  In _Proceedings of the 18th international joint conference on Artificial intelligence_ , pages 1011–1016, 2003.
  * [56] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman.  Data-efficient reinforcement learning with self-predictive representations.
  * Shelhamer et al. [2016] E. Shelhamer, P. Mahmoudieh, M. Argus, and T. Darrell.  Loss is its own reward: Self-supervision for reinforcement learning.  _arXiv preprint arXiv:1612.07307_ , 2016.
  * Shinners [2011] P. Shinners.  Pygame.  http://pygame.org/, 2011.
  * Shu et al. [2020] R. Shu, T. Nguyen, Y. Chow, T. Pham, K. Than, M. Ghavamzadeh, S. Ermon, and H. Bui.  Predictive coding for locally-linear control.  In _International Conference on Machine Learning_ , pages 8862–8871. PMLR, 2020.
  * Song and Ermon [2019] J. Song and S. Ermon.  Understanding the limitations of variational mutual information estimators.  _arXiv preprint arXiv:1910.06222_ , 2019.
  * Srinivas et al. [2020] A. Srinivas, M. Laskin, and P. Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  _arXiv preprint arXiv:2004.04136_ , 2020.
  * Still and Precup [2012] S. Still and D. Precup.  An information-theoretic approach to curiosity-driven reinforcement learning.  _Theory in Biosciences_ , 131(3):139–148, 2012\.
  * Stooke et al. [2020] A. Stooke, K. Lee, P. Abbeel, and M. Laskin.  Decoupling representation learning from reinforcement learning.  Technical report, UC Berkeley, 2020.
  * Sun et al. [2019] C. Sun, F. Baradel, K. Murphy, and C. Schmid.  Contrastive bidirectional transformer for temporal representation learning.  _arXiv preprint arXiv:1906.05743_ , 2019.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Thomas et al. [2017] V. Thomas, J. Pondard, E. Bengio, M. Sarfati, P. Beaudoin, M.-J. Meurs, J. Pineau, D. Precup, and Y. Bengio.  Independently controllable factors.  _arXiv preprint arXiv:1708.01289_ , 2017.
  * Van Roy and Dong [2019] B. Van Roy and S. Dong.  Comments on the du-kakade-wang-yang lower bounds.  _arXiv preprint arXiv:1911.07910_ , 2019.
  * Veeriah et al. [2019] V. Veeriah, M. Hessel, Z. Xu, J. Rajendran, R. L. Lewis, J. Oh, H. P. van Hasselt, D. Silver, and S. Singh.  Discovery of useful questions as auxiliary tasks.  In _Advances in Neural Information Processing Systems_ , pages 9306–9317, 2019.
  * Watter et al. [2015] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller.  Embed to control: A locally linear latent dynamics model for control from raw images.  In _Advances in neural information processing systems_ , pages 2746–2754, 2015.
  * Yu et al. [2019] T. Yu, G. Shevchuk, D. Sadigh, and C. Finn.  Unsupervised visuomotor control through distributional planning networks.  In _Robotics Science and Systems_ , 2019.
  * Zhan et al. [2020] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin.  A framework for efficient robotic manipulation.  _arXiv preprint arXiv:2012.07975_ , 2020.
  * Zhang et al. [2018a] A. Zhang, H. Satija, and J. Pineau.  Decoupling dynamics and reward for transfer learning.  _arXiv preprint arXiv:1804.10689_ , 2018a.
  * Zhang et al. [2018b] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine.  Solar: Deep structured latent representations for model-based reinforcement learning.  _arXiv preprint arXiv:1808.09105_ , 2018b.

##  Appendix A Appendix

###  A.1 Sufficiency of $\mathbb{J}_{fwd}$: Proof of Proposition 1

We describe the proofs for the sufficiency results from Section 5 here. We
begin by providing a set of lemmas, before proving the sufficiency of
$\mathbb{J}_{fwd}$.

Figure 8: Graphical model for Lemma 1, depicting true states $S$, states in
the representation $Z$, actions $A$, rewards $R$, and the variable $X$ (which
we will interpret as the sum of future rewards in the proof of Proposition 1).

######  Lemma 1.

Let $X$ be a random variable dependent on $S_{t+k}$, with the conditional
independence assumptions implied by the graphical model in Figure 8. (In the
main proof of Proposition 1, we will let $X$ be the sum of rewards from time
$t+k$ onwards.) If $I(Z_{t+k};Z_{t},A_{t})=I(S_{t+k};S_{t},A_{t})\forall k$,
then $I(X;Z_{t},A_{t})=I(X;S_{t},A_{t})\forall k$.

###### Proof.

Recall from Definition 1 that
$\phi_{\mathcal{Z}}(\mathbf{s})=p(Z|S=\mathbf{s})$. For proof by
contradiction, assume there is some $\phi_{\mathcal{Z}}$ and some $X$ such
that $I(X;Z_{t},A_{t})<I(X;S_{t},A_{t})$ and that
$I(Z_{t+k};Z_{t},A_{t})=I(S_{t+k};S_{t},A_{t})$. Now we know that because
$Z_{t}\rightarrow S_{t}\rightarrow S_{t+k}\rightarrow Z_{t+k}$ form a Markov
chain, by the data processing inequality (DPI) $I(Z_{t+k};Z_{t},A_{t})\leq
I(S_{t+k};Z_{t},A_{t})\leq I(S_{t+k};S_{t},A_{t})$. We will proceed by showing
that that $I(X;Z_{t},A_{t})<I(X;S_{t},A_{t})\implies
I(S_{t+k};Z_{t},A_{t})<I(S_{t+k};S_{t},A_{t})$, which gives the needed
contradiction.

Using chain rule, we can expand the following expression in two different
ways.

$I(X;Z_{t},S_{t},A_{t})=I(X;Z_{t}|S_{t},A_{t})+I(X;S_{t},A_{t})\\\
=0+I(X;S_{t},A_{t})$ (5)

$I(X;Z_{t},S_{t},A_{t})=I(X;S_{t}|Z_{t},A_{t})+I(X;Z_{t},A_{t})$ (6)

Note that the first term in Equation 5 is zero by the conditional independence
assumptions in Figure 8. Equating the expansions, we can see that to satisfy
our assumption that $I(X;Z_{t},A_{t})<I(X;S_{t},A_{t})$, we must have that
$I(X;S_{t}|Z_{t},A_{t})>0$.

Now we follow a similar procedure to expand the following expression:

$I(S_{t+k};Z_{t},S_{t},A_{t})=I(S_{t+k};Z_{t}|S_{t},A_{t})+I(S_{t+k};S_{t},A_{t})=0+I(S_{t+k};S_{t},A_{t})$
(7)

$I(S_{t+k};Z_{t},S_{t},A_{t})=I(S_{t+k};S_{t}|Z_{t},A_{t})+I(S_{t+k};Z_{t},A_{t})$
(8)

The first term in Equation 7 is zero by the conditional independence
assumptions in Figure 8. Comparing the first term in Equation 8 with the first
term in Equation 6, we see that because $S_{t}\rightarrow S_{t+k}\rightarrow
X$ form a Markov chain, by the DPI $I(S_{t+k};S_{t}|Z_{t},A_{t})\geq
I(X;S_{t}|Z_{t},A_{t})$. Therefore we must have
$I(S_{t+k};S_{t}|Z_{t},A_{t})>0$. Combining Equations 7 and 8:

$I(S_{t+k};S_{t},A_{t})=I(S_{t+k};S_{t}|Z_{t},A_{t})+I(S_{t+k};Z_{t},A_{t})$
(9)

Since $I(S_{t+k};S_{t}|Z_{t},A_{t})>0$,
$I(S_{t+k};Z_{t},A_{t})<I(S_{t+k};S_{t},A_{t})$, which is exactly the
contradiction we set out to show. ∎

######  Lemma 2.

If $I(Y;Z)=I(Y;X)$ and $Y\perp Z|X$, then $\forall
x,p(Y|X=x)=\mathbb{E}_{p(Y|Z)}p(Z|X=x)$.

###### Proof.

First note that the statement is not trivially true. Without any assumption
regarding MI, we can write,

$p(Y|X=x)=\int p(Y,Z|X=x)dz=\int p(Y|Z,X=x)p(Z|X=x)dz$ (10)

Comparing this with the statement we’d like to prove, we can see that the key
idea is to show that the MI equivalence implies that $p(Y|Z,X=x)=p(Y|Z)$.

Another way to write what we want to show is

$D_{KL}[p(Y|X)||\mathbb{E}_{p(Z|X)}p(Y|Z)]=0$ (11)

Our strategy will be to upper-bound this KL by a quantity that we will show to
be $0$. Since KL divergences are always lower-bounded $0$, this will prove
equality. We begin by writing out the definition of the KL divergence, and
then using Jensen’s inequality to upper-bound it.

$\begin{split}D_{KL}[p(Y|X)||\mathbb{E}_{p(Z|X)}p(Y|Z)]&=\mathbb{E}_{p(Y|X)}\log\left[\frac{p(Y|X)}{\mathbb{E}_{p(Z|X)}p(Y|Z)}\right]\\\
&=\mathbb{E}_{p(Y|X)}[\log p(Y|X)-\log\mathbb{E}_{p(Z|X)}p(Y|Z)]\\\
&\leq\mathbb{E}_{p(Y|X)}[\log p(Y|X)-\mathbb{E}_{p(Z|X)}\log
p(Y|Z)]\end{split}$ (12)

where the last inequality follows by Jensen’s inequality. Because KL
divergences are greater than $0$, this last expression is also greater than
$0$. Now let us try to relate $I(Y;Z)=I(Y;X)$ to this expression. We can re-
write this equality using the entropy definition of MI.

$H(Y)-H(Y|Z)=H(Y)-H(Y|X)$ (13)

The $H(Y)$ cancel and substituting the definition of entropy we have:

$\mathbb{E}_{p(Y,Z)}\log p(Y|Z)=\mathbb{E}_{p(Y,X)}\log p(Y|X)$ (14)

On the right-hand side, we can use the Tower property to re-write the
expectation as

$\mathbb{E}_{p(Y,X)}\log p(Y|X)=\mathbb{E}_{p(Z)}\mathbb{E}_{p(Y,X|Z)}\log
p(Y|X)=\mathbb{E}_{p(Y|X)p(X,Z)}\log p(Y|X)$ (15)

Now we can use the Tower property again to re-write the expectation on both
sides.

$\begin{split}\mathbb{E}_{p(X)}\mathbb{E}_{p(Y,Z|X)}\log
p(Y|Z)&=\mathbb{E}_{p(X)}\mathbb{E}_{p(Y|X)p(X,Z|X)}\log p(Y|X)\\\
\mathbb{E}_{p(X)}\mathbb{E}_{p(Y|X)p(Z|X)}\log
p(Y|Z)&=\mathbb{E}_{p(X)}\mathbb{E}_{p(Y|X)p(Z|X)}\log p(Y|X)\\\
\mathbb{E}_{p(X)}\mathbb{E}_{p(Y|X)p(Z|X)}[\log p(Y|X)&-\log p(Y|Z)]=0\\\
\mathbb{E}_{p(X)}\mathbb{E}_{p(Y|X)}[\log p(Y|X)&-\mathbb{E}_{p(Z|X)}\log
p(Y|Z)]=0\\\ \end{split}$ (16)

Compare this expression to Equation 12. Because Equation 12 is greater than
$0$, each term inside $\mathbb{E}_{p(X)}$ in Equation 16 is the same sign. If
the sum of elements that all have the same sign is zero, then each element is
zero. Therefore,

$0=\mathbb{E}_{p(Y|X)}[\log p(Y|X)-\mathbb{E}_{p(Z|X)}\log p(Y|Z)]\geq
D_{KL}[p(Y|X)||\mathbb{E}_{p(Z|X)}p(Y|Z)]$ (17)

Since the KL divergence is upper and lower-bounded by $0$, it must be equal to
$0$, and therefore $p(Y|X)=\mathbb{E}_{p(Y|Z)}p(Z|X)$ as we wanted to show. ∎

Given the lemmas stated above, we can then use them to prove the sufficiency
of $\mathbb{J}_{fwd}$.

######  Proposition 1.

(Sufficiency of $\mathbb{J}_{fwd}$) Let
$(\mathcal{S},\mathcal{A},\mathcal{T})$ be an MDP with dynamics
$p(S_{t+1}|S_{t},A_{t})$. Let the policy distribution $p(A|S)$ and steady-
state state occupancy $p(S)$ have full support on the action and state
alphabets $\mathcal{A}$ and $\mathcal{S}$ respectively. Let $\mathcal{R}$ be
the set of all reward functions that can be expressed as a function of the
state, $r:\mathcal{S}\rightarrow\mathbb{R}$. 222If the reward is a function of
the action, then $\mathbb{J}_{fwd}$ may not be sufficient. To see this, first
consider an MDP where the reward depends on a random variable in the state
that is independently sampled at each timestep. In this case,
$\mathbb{J}_{fwd}$ is still sufficient, because all policies will be equally
poor since the reward at the next timestep is completely unpredictable.
However, if the reward depends on the _action_ taken, then a policy trained on
the original state space could choose the high-reward action, but a
representation learned by $\mathbb{J}_{fwd}$ needn’t capture that information
if it doesn’t improve prediction of the next state. So in this latter case,
$\mathbb{J}_{fwd}$ can be insufficient. See Figure 8 for a graphical depiction
of the conditional independence relationships between variables.

For a representation $\phi_{\mathcal{Z}}$, if $I(Z_{t+1};Z_{t},A_{t})$ is
maximized $\forall t>0$ then $\forall r\in\mathcal{R}$ and
$\forall\mathbf{s}_{1},\mathbf{s}_{2}\in\mathcal{S}$,
$\phi_{\mathcal{Z}}(\mathbf{s}_{1})=\phi_{\mathcal{Z}}(\mathbf{s}_{2})\implies\forall\mathbf{a}\text{,}Q^{*}_{r}(\mathbf{a},\mathbf{s}_{1})=Q^{*}_{r}(\mathbf{a},\mathbf{s}_{2})$.

###### Proof.

Note that $(Z_{t+1};Z_{t},A_{t})$ is maximized if the representation
$\phi_{\mathcal{Z}}$ is taken to be the identity. In other words
$\max_{\phi}I(Z_{t+1};Z_{t},A_{t})=I(S_{t+1};S_{t},A_{t})$.

Define the random variable $\bar{R}_{t}$ to be the discounted return starting
from state $\mathbf{s}_{t}$.

$\bar{R}_{t}=\sum_{i=1}^{H-t}\gamma^{k}R_{t+i}$ (18)

Plug in $\bar{R}_{t}$ for the random variable $X$ in Lemma 1:

$I(Z_{t+1};Z_{t},A_{t})=I(S_{t+1};S_{t},A_{t})\qquad\implies\qquad
I(\bar{R}_{t+1};Z_{t},A_{t})=I(\bar{R}_{t+1};S_{t},A_{t})$ (19)

Now let $X=[S_{t},A_{t}]$, $Y=\bar{R}_{t}$, and $Z=Z_{t}$, and note that by
the structure of the graphical model in Figure 8, $Y\perp Z|X$. Plugging into
Lemma 2:

$\mathbb{E}_{p(\mathbf{z}_{t}|S_{t}=\mathbf{s})}p(\bar{R}_{t}|Z_{t},A_{t})=p(\bar{R}_{t}|S_{t}=\mathbf{s},A_{t})$
(20)

Now the $Q$-function given a reward function $r$ and a state-action pair
$(\mathbf{s},\mathbf{a})$ can be written as an expectation of this random
variable $\bar{R}_{t}$, given $S_{t}=\mathbf{s}$ and $A=\mathbf{a}$. (Note
that $p(\bar{R}_{t}|S_{t}=\mathbf{s},A_{t}=\mathbf{a})$ can be calculated from
the dynamics, policy, and reward distributions.)

$Q_{r}(\mathbf{s},\mathbf{a})=\mathbb{E}_{p(\bar{R}_{t}|S_{t}=\mathbf{s},A_{t}=\mathbf{a})}[\bar{R}_{t}]$
(21)

Since $\phi_{\mathcal{Z}}(\mathbf{s}_{1})=\phi_{\mathcal{Z}}(\mathbf{s}_{2})$,
$p(\mathbf{z}_{t}|S_{t}=\mathbf{s}_{1})=p(\mathbf{z}_{t}|S_{t}=\mathbf{s}_{2})$.
Therefore by Equation 20,
$p(\bar{R}_{t}|S_{t}=\mathbf{s}_{1},A_{t})=p(\bar{R}_{t}|S_{t}=\mathbf{s}_{2},A_{t})$.
Plugging this result into Equation 21,
$Q_{r}(\mathbf{a},\mathbf{s}_{1})=Q_{r}(\mathbf{a},\mathbf{s}_{2})$. Because
this reasoning holds for all $Q$-functions 333Note this result is stronger
than what we needed: it means that representations that maximize
$\mathbb{J}_{fwd}$ are guaranteed to be able to represent even sub-optimal
$Q$-functions. This makes sense in light of the fact that the proof holds for
all reward functions - the sub-optimal $Q$ under one reward is the optimal $Q$
under another., it also holds for the optimal $Q$, therefore
$Q^{*}_{r}(\mathbf{a},\mathbf{s}_{1})=Q^{*}_{r}(\mathbf{a},\mathbf{s}_{2})$.

∎

###  A.2 Experimental Details and Further Experiments

####  A.2.1 Didactic Experiments

The didactic examples are computed as follows. Given the list of states in the
MDP, we compute the possible representations, restricting our focus to
representations that group states into “blocks.” We do this because there are
infinite stochastic representations and the MI expressions we consider are not
convex in the parameters of $p(Z|S)$, making searching over these
representations difficult. Given each state representation, we compute the
value of the MI objective as well as the optimal value function using exact
value iteration. In these examples, we assume that the policy distribution is
uniform, and that the environment dynamics are deterministic. Since we
consider the infinite horizon setting, we use the steady-state state occupancy
in our calculations.

####  A.2.2 Deep RL Experiments

The deep RL experiments with the catcher game are conducted as follows. First,
we use a uniform random policy to collect $50$k transitions in the
environment. In this simple environment, the uniform random policy suffices to
visit all states (the random agent is capable of accidentally catching the
fruit, for example). Next, each representation learning objective is maximized
on this dataset. For all objectives, the images are pre-processed in the same
manner (resized to $64$x$64$ pixels and normalized) and embedded with a
convolutional network. The convolutional encoder consists of five
convolutional layers with ReLU activations and produces a latent vector with
dimension $256$. We use the latent vector to estimate each mutual information
objective, as described below.

Inverse information: We interpret the latent embeddings of the images $S_{t}$
and $S_{t+1}$ as the parameters of Gaussian distributions $p(Z|S_{t})$ and
$p(Z|S_{t+1})$. We obtain a single sample from each of these two
distributions, concatenate them and pass them through a single linear layer to
predict the action. The objective we maximize is the cross-entropy of the
predicted actions with the true actions, as in Agrawal et al. 2016 and
Shelhamer et al. 2016. To prevent recovering the trivial solution of
preserving all the information in the image, we add an information bottleneck
to the image embeddings. We tune the Lagrange multiplier on this bottleneck
such that the action prediction loss remains the same value as when trained
without the bottleneck. This approximates the objective
$\min_{\phi}I(Z;S)s.t.I_{inv}=\max I_{inv}$. To use the learned encoder for
RL, we embed the image from the current timestep and take the mean of the
predicted distribution as the state for the RL agent.

State-only information: We follow the Noise Contrastive Estimation (NCE)
approach presented in CPC (Oord et al. 2018). Denoting $Z_{t}$ and $Z{t+1}$ as
the latent embedding vectors from the convolutional encoders, we use a log-
bilinear model as in CPC to compute the score:
$f(Z_{t},Z_{t+1})=\exp(Z_{t}^{T}WZ_{t+1})$ for the cross-entropy loss. We also
experimented with an information bottleneck as described above, but found that
it wasn’t needed to obtain insufficient representations. To use the learned
encoder for RL, we embed the image from the current timestep and use this
latent vector as the state for the RL agent.

Forward information: We follow the same NCE strategy as for state-only
information, with the difference that we concatenate the action to $Z_{t}$
before computing the contrastive loss.

We then freeze the state encoder learned via MI-maximization and use the
representation as the state input for RL. The RL agent is trained using the
Soft Actor-Critic algorithm [25], modified for the discrete action
distribution (the Q-function outputs Q-values for all actions rather than
taking action as input, the policy outputs the action distribution directly
rather than parameters, and we can directly compute the expectation in the
critic loss rather than sampling). The policy and critic networks consist of
two hidden linear layers of $200$ units each with ReLU activations. We run $5$
random seeds of each experiment and plot the mean as a solid line and one
standard deviation as a shaded region.

Our implementation for the deep RL experiments is based on Garage [20] and we
use the default hyperparameters for the SAC algorithm found in that
repository. Garage is distributed under the MIT License. The environments we
use are from the pygame library [58] which is distributed under GNU LGPL
version 2.1. We used NVIDIA Titan X graphics cards to accelerate the training
of the representation learning algorithms with image inputs.

###  A.3 Alternative evaluation of the representation by predicting $Q^{*}$

Figure 9: Policy performance obtained from $Q$-functions trained to predict
$Q^{*}$ given state representations learned by each MI objective. Insufficient
objectives $\mathbb{J}_{inv}$ and $\mathbb{J}_{state}$ respectively perform
worse than sufficient objective $\mathbb{J}_{fwd}$.

In Section 6, we evaluated the learned representations by running a temporal
difference RL algorithm with the representation as the state input. In this
section, instead of using the bootstrap to learn the $Q$-function, we instead
regress the $Q$-function to the optimal $Q^{*}$. To do this, we first compute
the (roughly) optimal $Q^{*}$ by running RL with ground truth game state as
input and taking the learned $Q$ as $Q^{*}$. Then, we instantiate a new RL
agent and train it with the learned image representation as input, regressing
the $Q$-function directly onto the values of $Q^{*}$. We evaluate the policy
derived from this new $Q$-function, and plot the results for both the catcher
and catcher-grip environments in Figure 9. We find that similar to the result
achieved using the bootstrap, the policy performs poorly when using
representations learned by insufficient objectives ($\mathbb{J}_{inv}$ in
catcher and $\mathbb{J}_{state}$ in catcher-grip). Interestingly, we find that
the error between the learned $Q$-values and the $Q^{*}$-values is roughly the
same for sufficient and insufficient representations. We hypothesize that this
discrepancy between $Q$-value error and policy performance is due to the fact
that small differences in $Q$-values on a small set of states can result in
significant behavior differences in the policy.

###  A.4 Changing the data distribution used for representation learning

In Section 6, we collect the dataset used to optimize the representation
learning objectives with a uniform random policy. Here, we perform an ablation
where we collect this dataset using the optimal policy, trained from ground
truth state (agent and fruit positions). We show the results for the catcher
environment in Figure 11. As predicted by our theoretical analysis,
$\mathbb{J}_{inv}$ is still insufficient, and can fail to represent the fruit,
leading to poor performance of the RL agent.

###  A.5 Temporally correlated visual distractors

In this section, we perform experiments with temporally correlated background
distractors by animating the colored circles (see Figure 6) to bounce around
the screen. We show results for the catcher environment in Figure 11. The
results are similar to those in Figure 7 of the paper - the insufficient
objective $\mathbb{J}_{inv}$ results in representations poorly suited to
learning the task, while $\mathbb{J}_{fwd}$ yields a useful representation.
However, the $\mathbb{J}_{fwd}$ objective could suffer if the complexity of
the background were further increased. With limited model capacity,
$\mathbb{J}_{fwd}$ might fixate on the moving background rather than the task-
relevant fruit, because it is incentivised to predict all the state
information that is predictable. Regardless of the background complexity,
$\mathbb{J}_{inv}$ is still insufficient in this MDP.

Figure 10: Performance of RL agents trained with state representations learned
with data collected by the optimal policy, catcher environment.

Figure 11: Performance of RL agents learned from state representations when
observations contain visual distractors that are correlated across timesteps,
catcher environment.


# Scientific Literature Text Mining and the Case for Open Access

Gopal P. Sarma1
1\. _School of Medicine, Emory University, Atlanta, GA USA_
Email<EMAIL_ADDRESS>
## I. Introduction

The growth of institutional science following the Second World War has
resulted in a range of unanticipated infrastructural problems, ranging from
the overproduction of PhDs relative to the number of faculty positions,
protracted educational trajectories for many aspiring scientists, and most
alarmingly, a “reproducibility crisis,” whereby the veracity of large subsets
of the research literature has been called into question [1, 2, 3, 4, 5, 6, 7,
8, 9].

A significant target of institutional reform to address the larger set of
issues created by dramatic scientific growth has been the academic publishing
model. Access to scholarly output has traditionally been restricted to wealthy
universities, whose library systems are charged exorbitant fees to maintain
annual subscriptions. In contrast, an “open access” model is one in which
research articles are made freely available to all, largely taking advantage
of the infrastructural efficiencies provided by the Internet. Indeed, many new
journals are online only and do not distribute printed copies of their
collections.

Traditionally, there have been two primary arguments for open access
publishing. The first is maximizing the accessibility of research output.
Subscriptions to the top journals can be prohibitively expensive and only the
wealthiest universities and industrial research labs can afford them. Removing
the barrier to access opens the possibility for novel results to gain
significantly greater exposure and scrutiny, particularly in countries with
developing scientific infrastructures that have to be thrifty with resource
allocation. The same is true for smaller universities or research-oriented
companies for whom annual subscriptions for a full spectrum of journals cannot
be reasonably budgeted.

The second argument for open access publishing is eliminating the “double-
billing effect” of publicly funded research. Surely tax-paying citizens should
not have to pay twice to read the output of research that they have already
contributed to funding. Indeed, the Public Access of Policy of the National
Institutes of Health now requires that publicly funded research be made freely
available via BioMed Central within 12 months of publication [10].

An important and overlooked argument for open-access publishing is _scientific
literature text mining_ , i.e. data science efforts which treat the scientific
corpus itself as a massive dataset to analyze and extract important,
actionable insights. In the remainder of the article, I give a brief
discussion of scientific literature text mining, its relationship to open
access publishing and the “reproducibility crisis.” I close with a call to all
scholars to prioritize publication in journals that provide complete,
unrestricted access to research articles and to support the legal availability
of bulk-access to scientific papers as well as research efforts that make use
of them.

## II. Scientific Literature Text Mining

I use the phrase _scientific literature text mining_ to refer to data analysis
of the scientific corpus, rather than the data sets that are produced by
research studies. One can think of scientific literature text mining as
representing a full-fledged generalization of review articles, systematic
reviews, and meta-analyses whereby sophisticated tools from the modern data
science toolkit are utilized to extract novel insights from the scientific
corpus itself.

The applications of data science to the scientific corpus is in its nascent
stages. It has the potential to advance our understanding of global scientific
trends, the relationship between fundamental research and technological
development, and fraud detection, to name just a few possible applications
(see for example, [11, 12, 13, 14, 15, 16, 17, 18, 19, 20])111There has been
slow, but steady growth of research in recent years utilizing data science to
investigate trends in the origin, growth, and dissemination of knowledge?the
references given above are simply a sample of contemporary work. See, for
example, research at The Santa Fe Institute, The Knowledge Lab at the
University of Chicago, The National Center for Text Mining, and the research
group of Peter Murray-Rust for full-fledged efforts in scientific literature
text mining. See also OpenCitations for an open repository of citation data
and NeuroSynth and NeuroElectro for successfully implemented text mining
platforms in the neurosciences..

Performing such analyses requires unrestricted access to the research
literature so that articles can themselves can be treated as data sets to be
used as part of a data science pipeline. Therefore, publishing companies which
have released articles under proprietary formats, while complying with a
narrow interpretation of open access, are preventing the development of a
powerful set of tools and cultural practices for advancing science. Scientific
literature text mining is particularly important in the context of the
“reproducibility crisis.” Indeed, in recent years, significant attention has
been drawn to low rates of reproducibility of research studies across a number
of disciplines. From reproducibility initiatives, to re-examining the
incentive structures of academic research, to novel journalistic practices
such as post-publication peer review, the reproducibility crisis has been a
significant source of controversy, discussion, and institutional action [4, 5,
7, 8, 21, 22, 23, 24, 25, 26, 27].

However, we are barely beginning to understand the scope of the problem. For
example, the studies which uncovered a large number of irreproducible results
were conducted in a limited range of subjects, and we cannot generalize from
these studies to understand what the “reproducibility distribution” looks like
for the entirety of science. While discussions surrounding the reproducibility
crisis have largely focused on basic research in the biological sciences,
social sciences and psychology, and clinical medicine, the shared incentive
structures of academic research suggests that we should view all research with
a more critical eye—including the engineering and applied sciences—until the
full scope of the reproducibility crisis has been more thoroughly
characterized. Furthermore, as the applied sciences depend on the veracity of
results producing by allied fields in basic research, these subjects are not
shielded from problematic research conducted upstream in the research
pipeline. Indeed, some of the most significant results in characterizing the
scope of the problems in the biomedical sciences have come from the
pharmaceutical industry, which relies on published drug targets as a
foundational element of drug design [21, 22]. Scientific literature text
mining has the potential to play a key role in more accurately characterizing
the status of different fields by identifying “linchpin results,” which would
be of particularly high value to be the focus of targeted replication studies.
For instance, by examining citation networks, it may be possible to identify a
small subset of candidate papers that can be directly examined by specialists
in a field to determine if adequate sample sizes or appropriate statistical
tests were used.

The possibilities for exploratory data analysis are endless. We might imagine
using natural language processing and textual analysis to characterize the
transference of ideas between fields, the emergence of new concepts, and the
transition from basic to applied research. Techniques such as these will also
allow us to develop more refined methodologies for characterizing the
importance of individual contributions and shift away from much abused metrics
such as the impact factor. For example, by using predictive models or
collaborative filtering, a fully digitized corpus should allow for articles to
have citations automatically generated, in addition to the manually added
citations by the authors themselves. These citations could be continually
updated in real time as the models are refined or new research articles are
published. This would allow the notion of citations to be extended to include
future results as well as those that authors were aware of at the time of
publication. However, none of these tantalizing possibilities can be realized
if restrictions are placed on access to the research literature in bulk
form.222The move towards a pre-print model for academic publications is a
positive development for scientific literature text mining. Long practiced in
the physics community via the arXiv pre-print server, and steadily being
adopted by other fields, the pre-print model allows for drafts of publications
to be immediately available online prior to submission to a journal. arXiv,
originally launched by Los Alamos National Laboratory and now overseen by the
Cornell University Library System, has made the full source code and PDFs of
all its pre-prints available for free via Amazon Web Services [28]. The
availability of this corpus is a powerful resource for scientific literature
text mining. However, to date, it is the only pre-print server to do so and
newer repositories should also follow in its footsteps.
As the pre-print model itself becomes more widely adopted, the simultaneous
availability of both pre-prints and the final journal publication will allow
for critical analysis of peer review, a facet of the modern scientific process
[29, 30, 31] whose re-examination is essential in addressing the
reproducibility crisis. For example, simple textual analysis of pre-prints and
their published counterparts will allow us to characterize the extent to which
peer review influences manuscripts, and how level of influence varies across
different subjects.  It is simply inadequate for articles to be released in
proprietary formats that cannot be processed by automated tools.

Therefore, I argue that one of the primary motivations for open access
publishing is the _enormous benefit to science and society of scientific
literature text mining_. We need a scientific analogue to CommonCrawl, an open
repository of scientific articles for use in exploratory data analysis.
Ironically, this argument is not new, and indeed, was anticipated as part of
the original definition of open access given at the Budapest Open Access
Initiative:

> By “open access” to this literature, we mean its free availability on the
> public internet, permitting any users to read, download, copy, distribute,
> print, search, or link to the full texts of these articles, crawl them for
> indexing, pass them as data to software, or use them for any other lawful
> purpose, without financial, legal, or technical barriers other than those
> inseparable from gaining access to the internet itself. The only constraint
> on reproduction and distribution, and the only role for copyright in this
> domain, should be to give authors control over the integrity of their work
> and the right to be properly acknowledged and cited [32].

One of the key phrases in this definition is allowing users to “pass the
articles as data to software,” that is, _to treat the scientific corpus itself
as data._ However, when these words were first written, the phrase “data
science” had yet to be coined, and the enormous growth of the field, largely
driven by social media, had yet to take place. The original framers of the
definition of open access had the vision and foresight to anticipate that
unrestricted access to the research literature should include far more than
the ability for individuals to freely read scholarly articles. They should
also be able to conduct sophisticated analyses of large bodies of literature
using computational techniques that have only become possible in recent years.

## III. Conclusion

Realizing the full vision of scientific literature text mining requires
unrestricted access to the scientific corpus. We should aspire to build a
fully open infrastructure where there are APIs for every journal and pre-print
repository, allowing anyone to access the data and metadata for every article
and conduct exploratory or targeted data analyses. Taking advantage of a fully
digitized and easily accessible corpus of knowledge, data scientists will
build information dashboards providing intuitive insights into an increasingly
complex knowledge base. Most importantly, scientific literature text mining
may come to play a crucial role in addressing the “reproducibility crisis” by
enabling analyses of large corpuses of scientific papers to uncover “linchpin
results” which would subsequently be the focus of targeted replication
efforts.

Scholars from all disciplines should be aware of the enormous benefit to
society at large of scientific literature text mining and to prioritize
publication in journals that allow for complete, unrestricted access to their
articles. Furthermore, all researchers should lend their support to making the
scientific literature legally available in bulk form and encourage data
science efforts which advance this important and novel component of the modern
research enterprise.

## Acknowledgments

I would like to thank Travis Rich and Daniel Weissman for insightful
discussions and feedback on the manuscript. A special thanks to Devin Berg and
the editorial staff of _The Journal of Open Engineering_ for starting an
important and innovative journal and the reviewers Titus Brown and Shreejoy
Tripathy for their feedback and willingness to participate in an open review
process.

## ORCID

Gopal P. Sarma 0000-0002-9413-6202

## References

  * [1] H. Bode, F. Mosteller, J. Tukey, and C. Winsor, “The Education of a Scientific Generalist,” Science, vol. 109, no. 2840, pp. 553–558, 1949\.
  * [2] V. Bush, “Science: The Endless Frontier,” Science Education, vol. 29, no. 4, pp. 218–218, 1945.
  * [3] V. Narayanamurti, T. Odumosu, and L. Vinsel, “RIP: The Basic/Applied Research Dichotomy,” Issues in Science and Technology, vol. 29, no. 2, p. 31, 2013.
  * [4] J. P. A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Med, vol. 2, p. e124, 08 2005.
  * [5] W. Gunn, “Reproducibility: fraud is not the big problem,” Nature, vol. 505, no. 7484, pp. 483–483, 2014.
  * [6] D. Adam and J. Knight, “Journals under pressure: Publish, and be damned…,” Nature, vol. 419, no. 6909, pp. 772–776, 2002.
  * [7] E. Check and D. Cyranoski, “Korean scandal will have global fallout,” Nature, vol. 438, no. 7071, pp. 1056–1057, 2005.
  * [8] R. Horton, “What’s medicine’s 5 sigma?,” The Lancet, vol. 385, no. 9976, 2015.
  * [9] B. Alberts, M. W. Kirschner, S. Tilghman, and H. Varmus, “Rescuing US biomedical research from its systemic flaws,” Proceedings of the National Academy of Sciences, vol. 111, no. 16, pp. 5773–5777, 2014.
  * [10] “NIH Open Access Policy.” https://publicaccess.nih.gov/policy.htm.  Accessed: 2016-10-14.
  * [11] D. M. Markowitz and J. T. Hancock, “Linguistic Obfuscation in Fraudulent Science,” Journal of Language and Social Psychology, p. 0261927X15614605, 2015.
  * [12] Y. Ding, “Applying weighted PageRank to author citation networks,” Journal of the American Society for Information Science and Technology, vol. 62, no. 2, pp. 236–245, 2011.
  * [13] Y. Ding, “Topic-based PageRank on author cocitation networks,” Journal of the American Society for Information Science and Technology, vol. 62, no. 3, pp. 449–466, 2011.
  * [14] Y. Ding, X. Liu, C. Guo, and B. Cronin, “The distribution of references across texts: Some implications for citation analysis,” Journal of Infometrics, vol. 7, no. 3, pp. 583–592, 2013.
  * [15] W. Zhu and J. Guan, “A bibliometric study of service innovation research: based on complex network analysis,” Scientometrics, vol. 94, no. 3, pp. 1195–1216, 2013.
  * [16] X. Zhu, P. Turney, D. Lemire, and A. Vellino, “Measuring academic influence: Not all citations are equal,” Journal of the Association for Information Science and Technology, vol. 66, no. 2, pp. 408–427, 2015.
  * [17] M. Song, S. Kim, G. Zhang, Y. Ding, and T. Chambers, “Productivity and influence in bioinformatics: A bibliometric analysis using PubMed central,” Journal of the Association for Information Science and Technology, vol. 65, no. 2, pp. 352–371, 2014.
  * [18] S. Valverde, R. V. Solé, M. A. Bedau, and N. Packard, “Topology and evolution of technology innovation networks,” Physical Review E, vol. 76, no. 5, p. 056118, 2007.
  * [19] B. Gress, “Properties of the USPTO patent citation network: 1963–2002,” World Patent Information, vol. 32, no. 1, pp. 3–21, 2010.
  * [20] R. V. Solée, S. Valverde, M. R. Casals, S. A. Kauffman, D. Farmer, and N. Eldredge, “The evolutionary ecology of technological innovations,” Complexity, vol. 18, no. 4, pp. 15–27, 2013.
  * [21] P. Campbell, ed., Challenges in Irreproducible Research, vol. 526, Nature Publishing Group, 2015.
  * [22] F. Prinz, T. Schlange, and K. Asadullah, “Believe it or not: how much can we rely on published data on potential drug targets?,” Nature Reviews Drug Discovery, vol. 10, no. 712, 2011.
  * [23] Editors, “Trouble at the lab,” The Economist, 10 2013.
  * [24] Neuroskeptic, “Reproducibility Crisis: The Plot Thickens,” Discover Magazine, 10 2015.
  * [25] B. Carey, “Science, Now Under Scrutiny Itself,” The New York Times, 7 2015\.
  * [26] K. M. Palmer, “Psychology is in a Crisis Over Whether It’s in a Crisis,” Wired Magazine, 3 2016.
  * [27] J. S. Flier, “How to Keep Bad Science From Getting Into Print,” The Wall Street Journal, 3 2016.
  * [28] “arXiv Bulk Data Access.” https://arxiv.org/help/bulk_data_s3.  Accessed: 2016-10-14.
  * [29] D. A. Kronick, “Peer review in 18th-century scientific journalism,” Jama, vol. 263, no. 10, pp. 1321–1322, 1990.
  * [30] J. C. Burnham, “The evolution of editorial peer review,” Jama, vol. 263, no. 10, pp. 1323–1329, 1990.
  * [31] R. Spier, “The history of the peer-review process,” Trends in Biotechnology, vol. 20, no. 8, pp. 357–358, 2002.
  * [32] “Budapest Open Access Initiative.” http://www.budapestopenaccessinitiative.org.  Accessed: 2016-10-14.