learning to maze navigation problems, we present an end-to-end deep
reinforcement learning approach that can be applied on a city scale.
Recognising that successful navigation relies on integration of general
policies with locale-specific knowledge, we propose a dual pathway
architecture that allows locale-specific features to be encapsulated, while
still enabling transfer to multiple cities. We present an interactive
navigation environment that uses Google StreetView for its photographic
content and worldwide coverage, and demonstrate that our learning method
allows agents to learn to navigate multiple cities and to traverse to target
destinations that may be kilometres away. A video summarizing our research and
showing the trained agent in diverse city environments as well as on the
transfer task is available at: https://sites.google.com/view/streetlearn.,
archivePrefix = arXiv, arxivId = 1804.00168, author = Mirowski, Piotr and
Grimes, Matthew Koichi and Malinowski, Mateusz and Hermann, Karl Moritz and
Anderson, Keith and Teplyashin, Denis and Simonyan, Karen and Kavukcuoglu,
Koray and Zisserman, Andrew and Hadsell, Raia, eprint = 1804.00168, file = ::,
title = Learning to Navigate in Cities Without a Map, url =
http://arxiv.org/abs/1804.00168, year = 2018 @articleHunt1992, abstract = This
paper focuses on the promise of artificial neural networks in the realm of
modelling, identification and control of nonlinear systems. The basic ideas
and techniques of artificial neural networks are presented in language and
notation familiar to control engineers. Applications of a variety of neural
network architectures in control are surveyed. We explore the links between
the fields of control science and neural networks in a unified presentation
and identify key areas for future research. 1992., archivePrefix = arXiv,
arxivId = 0005-1098/92 $5.00 + 0.00, author = Hunt, K. J. and Sbarbaro, D. and
Zbikowski, R. and Gawthrop, P. J., doi = 10.1016/0005-1098(92)90053-I, eprint
= 92 $5.00 + 0.00, file = ::, isbn = ISSN~~0005-1098, issn = 00051098, journal
= Automatica, keywords = Neural networks,nonlinear control systems,nonlinear
systems modelling,systems identification, number = 6, pages = 1083–1112,
primaryClass = 0005-1098, title = Neural networks for control systems-A
survey, volume = 28, year = 1992 @inproceedingskaelbling1993goals, abstract =
Temporal diierence methods solve the tem-poral credit assignment problem for
reinforce-ment learning. An important subproblem of general reinforcement
learning is learning to achieve dynamic goals. Although existing tem-poral
diierence methods, such as Q learning, can be applied to this problem, they do
not take advantage of its special structure. This pa-per presents the DG-
learning algorithm, which learns eeciently to achieve dynamically chang-ing
goals and exhibits good knowledge transfer between goals. In addition, this
paper shows how traditional relaxation techniques can be applied to the
problem. Finally, experimen-tal results are given that demonstrate the su-
periority of DG learning over Q learning in a moderately large, synthetic,
non-deterministic domain., author = Kaelbling, L P, booktitle = International
Joint Conference on Artificial Intelligence (IJCAI), file = ::, keywords =
learning (artificial intelligence),temporal reason, pages = 1094 – 8, title =
Learning to achieve goals, volume = vol.2, year = 1993 @articleCheney2013,
abstract = In 1994 Karl Sims showed that computational evolution can produce
interesting morphologies that resemble natural or-ganisms. Despite nearly two
decades of work since, evolved morphologies are not obviously more complex or
natural, and the field seems to have hit a complexity ceiling. One hypothesis
for the lack of increased complexity is that most work, including Sims’,
evolves morphologies composed of rigid elements, such as solid cubes and
cylinders, limiting the design space. A second hypothesis is that the encod-
ings of previous work have been overly regular, not allow-ing complex
regularities with variation. Here we test both hypotheses by evolving soft
robots with multiple materials and a powerful generative encoding called a
compositional pattern-producing network (CPPN). Robots are selected for
locomotion speed. We find that CPPNs evolve faster robots than a direct
encoding and that the CPPN morphologies appear more natural. We also find that
locomotion per-formance increases as more materials are added, that di-versity
of form and behavior can be increased with diâµer-ent cost functions without
stifling performance, and that organisms can be evolved at diâµerent levels
of resolution. These findings suggest the ability of generative soft-voxel
systems to scale towards evolving a large diversity of com-plex, natural,
multi-material creatures. Our results suggest that future work that combines
the evolution of CPPN-encoded soft, multi-material robots with modern
diversity-encouraging techniques could finally enable the creation of
creatures far more complex and interesting than those pro-duced by Sims nearly
twenty years ago., author = Cheney, Nick and Maccurdy, Robert and Clune, Jeff
and Lipson, Hod, file = ::, keywords = Algorithms,CPPN-NEAT,Categories and
Subject Descriptors,Design,Evolving Morphologies,Experimentation
Keywords,Generative Encodings,Genetic Algorithms,HyperNEAT,I211 [Distributed
Artificial Intelligence],Intelligent Agents General Terms,Soft-Robotics, title
= Unshackling Evolution: Evolving Soft Robots with Multiple Materials and a
Powerful Generative Encoding, url =
http://jeffclune.com/publications/2013_Softbots_GECCO.pdf, year = 2013
@articleLevine2018, abstract = The framework of reinforcement learning or
optimal control provides a mathe-matical formalization of intelligent decision
making that is powerful and broadly applicable. While the general form of the
reinforcement learning problem enables effective reasoning about uncertainty,
the connection between reinforcement learn-ing and inference in probabilistic
models is not immediately obvious. However, such a connection has considerable
value when it comes to algorithm design: for-malizing a problem as
probabilistic inference in principle allows us to bring to bear a wide array
of approximate inference tools, extend the model in flexible and powerful
ways, and reason about compositionality and partial observability. In this
article, we will discuss how a generalization of the reinforcement learning or
optimal control problem, which is sometimes termed maximum entropy rein-
forcement learning, is equivalent to exact probabilistic inference in the case
of deterministic dynamics, and variational inference in the case of stochastic
dynam-ics. We will present a detailed derivation of this framework, overview
prior work that has drawn on this and related ideas to propose new
reinforcement learning and control algorithms, and describe perspectives on
future research., archivePrefix = arXiv, arxivId = arXiv:1805.00909v2, author
= Levine, Sergey, eprint = arXiv:1805.00909v2, file = ::, title =
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and
Review, url = https://arxiv.org/pdf/1805.00909.pdf, year = 2018
@articleLesort2018, abstract = Representation learning algorithms are designed
to learn abstract features that character-ize data. State representation
learning (SRL) focuses on a particular kind of representation learning where
learned features are in low dimension, evolve through time, and are influenced
by actions of an agent. As the representation learned captures the variation
in the environment generated by agents, this kind of representation is
particularly suitable for robotics and con-trol scenarios. In particular, the
low dimension helps to overcome the curse of dimensionality, provides easier
interpretation and utilization by humans and can help improve performance and
speed in policy learning algorithms such as reinforcement learning. This
survey aims at covering the state-of-the-art on state representation learning
in the most recent years. It reviews different SRL methods that involve
interaction with the envi-ronment, their implementations and their
applications in robotics control tasks (simulated or real). In particular, it
highlights how generic learning objectives are differently exploited in the
reviewed algorithms. Finally, it discusses evaluation methods to assess the
representation learned and summarizes current and future lines of research.,
archivePrefix = arXiv, arxivId = arXiv:1802.04181v1, author = Lesort, Timothée
and Díaz-Rodríguez, Natalia and Goudou, Jean-François and Filliat, David,
eprint = arXiv:1802.04181v1, file = ::, keywords = Disentanglement of control
factors,Learning Disentangled Representations,Low Dimensional Embedding
Learning,Reinforcement Learn-ing,Robotics,State Representation Learning, title
= State Representation Learning for Control: An Overview, url =
https://arxiv.org/pdf/1802.04181.pdf, year = 2018 @articleKonda, abstract = We
present an approach to predicting velocity and direction changes from visual
information (” visual odom-etry ”) using an end-to-end, deep learning-based
architecture. The architecture uses a single type of compu-tational module and
learning rule to extract visual motion, depth, and finally odometry
information from the raw data. Representations of depth and motion are
extracted by detecting synchrony across time and stereo channels using network
layers with multiplicative interactions. The extracted representations are
turned into information about changes in velocity and direction using a
convolutional neural network. Preliminary results show that the architecture
is capable of learning the resulting mapping from video to egomotion., author
= Konda, Kishore and Memisevic, Roland, file = ::, keywords = Convolutional
networks,Motion,Stereo,Visual odometry, title = Learning visual odometry with
a convolutional network, url = http://www.iro.umontreal.ca/
memisevr/pubs/VISAPP2015.pdf @articleCoates, abstract = We consider the
problem of learning to follow a desired trajectory when given a small num-ber
of demonstrations from a sub-optimal ex-pert. We present an algorithm that (i)
ex-tracts theâinitially unknownâdesired tra-jectory from the sub-optimal
expert’s demon-strations and (ii) learns a local model suit-able for control
along the learned trajectory. We apply our algorithm to the problem of
autonomous helicopter flight. In all cases, the autonomous helicopter’s
performance ex-ceeds that of our expert helicopter pilot’s demonstrations.
Even stronger, our results significantly extend the state-of-the-art in au-
tonomous helicopter aerobatics. In particu-lar, our results include the first
autonomous tic-tocs, loops and hurricane, vastly superior performance on
previously performed aero-batic maneuvers (such as in-place flips and rolls),
and a complete airshow, which requires autonomous transitions between these
and various other maneuvers., author = Coates, Adam and Abbeel, Pieter and Ng,
Andrew Y, file = ::, title = Learning for Control from Multiple
Demonstrations, url = https://www-
cs.stanford.edu/people/ang/papers/icml08-LearningForControlFromMultipleDemonstrations.pdf
@inproceedingswe2018mpo, author = Abdolmaleki, Abbas and Springenberg, Jost
Tobias and Tassa, Yuval and Munos, Remi and Heess, Nicolas and Riedmiller,
Martin, booktitle = International Conference on Learning Representations
(ICLR), file = ::, pages = 1–19, title = Maximum a Posteriori Policy
Optimisation, year = 2018 @articlefinn2017deepvf, abstract = A key challenge
in scaling up robot learning to many skills and environments is removing the
need for human supervision, so that robots can collect their own data and
improve their own performance without being limited by the cost of requesting
human feedback. Model-based reinforcement learning holds the promise of
enabling an agent to learn to predict the effects of its actions, which could
provide flexible predictive models for a wide range of tasks and environments,
without detailed human supervision. We develop a method for combining deep
action-conditioned video prediction models with model-predictive control that
uses entirely unlabeled training data. Our approach does not require a
calibrated camera, an instrumented training setup , nor precise sensing and
actuation. Our results show that our method enables a real robot to perform
nonprehensile manipulation-pushing objects-and can handle novel objects not
seen during training., author = Chelsea Finn and Sergey Levine, journal =
ICRA, title = Deep Visual Foresight for Planning Robot Motion, year = 2017,
@inproceedingsichter2019robot, author = Brian Ichter and Marco Pavone, journal
= ICRA, title = Robot Motion Planning in Learned Latent Spaces, year = 2019,
@articleichter2018learning, abstract = A defining feature of sampling-based
motion planning is the reliance on an implicit representation of the state
space, which is enabled by a set of probing samples. Traditionally, these
samples are drawn either probabilistically or deterministically to uniformly
cover the state space. Yet, the motion of many robotic systems is often
restricted to ’small’ regions of the state space, due to e.g. differential
constraints or collision-avoidance constraints. To accelerate the planning
process, it is thus desirable to devise non-uniform sampling strategies that
favor sampling in those regions where an optimal solution might lie. This
paper proposes a methodology for nonuniform sampling, whereby a sampling
distribution is learned from demonstrations, and then used to bias sampling.
The sampling distribution is computed through a conditional variational
autoencoder, allowing sample generation from the latent space conditioned on
the specific planning problem. This methodology is general, can be used in
combination with any sampling-based planner, and can effectively exploit the
underlying structure of a planning problem while maintaining the theoretical
guarantees of sampling-based approaches. Specifically, on several planning
problems, the proposed methodology is shown to effectively learn
representations for the relevant regions of the state space, resulting in an
order of magnitude improvement in terms of success rate and convergence to the
optimal cost., author = Brian Ichter and James Harrison and Marco Pavone, doi
= 10.1109/ICRA.2018.8460730, isbn = 9781538630815, issn = 10504729, journal =
ICRA, month = 9, pages = 7087-7094, publisher = Institute of Electrical and
Electronics Engineers Inc., title = Learning Sampling Distributions for Robot
Motion Planning, year = 2018,

@articlecoreyes2018sectar, abstract = In this work, we take a representation
learning perspective on hierarchical reinforcement learning, where the problem
of learning lower layers in a hierarchy is transformed into the problem of
learning trajectory-level generative models. We show that we can learn
continuous latent representations of trajectories, which are effective in
solving temporally extended and multi-stage problems. Our proposed model,
SeCTAR, draws inspiration from variational autoencoders, and learns latent
representations of trajectories. A key component of this method is to learn
both a latent-conditioned policy and a latent-conditioned model which are
consistent with each other. Given the same latent, the policy generates a
trajectory which should match the trajectory predicted by the model. This
model provides a built-in prediction mechanism, by predicting the outcome of
closed loop policy behavior. We propose a novel algorithm for performing
hierarchical RL with this model, combining model-based planning in the learned
latent space with an unsupervised exploration objective. We show that our
model is effective at reasoning over long horizons with sparse rewards for
several simulated tasks, outperforming standard reinforcement learning methods
and prior methods for hierarchical reasoning, model-based planning, and
exploration., author = John D. Co-Reyes and Yu Xuan Liu and Abhishek Gupta and
Benjamin Eysenbach and Pieter Abbeel and Sergey Levine, isbn = 9781510867963,
journal = 35th International Conference on Machine Learning, ICML 2018, pages
= 1637-1647, publisher = International Machine Learning Society (IMLS), title
= Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning
with Trajectory Embeddings, volume = 3, year = 2018, @articleSilver, abstract
= One of the key challenges of artificial intelli-gence is to learn models
that are effective in the context of planning. In this document we intro-duce
the predictron architecture. The predictron consists of a fully abstract
model, represented by a Markov reward process, that can be rolled for-ward
multiple ” imagined ” planning steps. Each forward pass of the predictron
accumulates in-ternal rewards and values over multiple plan-ning depths. The
predictron is trained end-to-end so as to make these accumulated values ac-
curately approximate the true value function. We applied the predictron to
procedurally generated random mazes and a simulator for the game of pool. The
predictron yielded significantly more accurate predictions than conventional
deep neu-ral network architectures., author = Silver, David and Hasselt, Hado
Van and Hessel, Matteo and Schaul, Tom and Guez, Arthur and Harley, Tim and
Dulac-Arnold, Gabriel and Reichert, David and Rabinowitz, Neil and Barreto,
Andre and Degris, Thomas, file = ::, title = The Predictron: End-To-End
Learning and Planning, url = https://arxiv.org/pdf/1612.08810.pdf
@articleRitchie, abstract = Probabilistic programming languages (PPLs) are a
powerful modeling tool, able to represent any computable probability
distribution. Unfortunately, probabilistic program inference is often
intractable, and existing PPLs mostly rely on expensive, approximate sampling-
based methods. To alleviate this problem, one could try to learn from past
inferences, so that future inferences run faster. This strategy is known as
amortized inference; it has recently been applied to Bayesian net-works [28,
22] and deep generative models [20, 15, 24]. This paper proposes a system for
amortized inference in PPLs. In our system, amortization comes in the form of
a parameterized guide program. Guide programs have similar structure to the
original program, but can have richer data flow, including neural network
components. These networks can be optimized so that the guide approximately
samples from the posterior distribution defined by the original program. We
present a flexible interface for defining guide programs and a stochastic
gradient-based scheme for optimizing guide parameters, as well as some
preliminary results on automatically deriving guide programs. We explore in
detail the common machine learning pattern in which a ’local’ model is
specified by ’global’ random values and used to generate independent observed
data points; this gives rise to amortized local inference supporting global
model learning., author = Ritchie, Daniel and Horsfall, Paul and Goodman, Noah
D, file = ::, title = Deep Amortized Inference for Probabilistic Programs, url
= https://arxiv.org/pdf/1610.05735.pdf @articleBlei2017, abstract = One of the
core problems of modern statistics is to approximate difficult-to-compute
probability densities. This problem is especially important in Bayesian
statistics, which frames all inference about unknown quantities as a
calculation involving the posterior density. In this paper, we review
variational inference (VI), a method from machine learning that approximates
probability densities through optimization. VI has been used in many
applications and tends to be faster than classical methods, such as Markov
chain Monte Carlo sampling. The idea behind VI is to first posit a family of
densities and then to find the member of that family which is close to the
target. Closeness is measured by Kullback-Leibler divergence. We review the
ideas behind mean-field variational inference, discuss the special case of VI
applied to exponential family models, present a full example with a Bayesian
mixture of Gaussians, and derive a variant that uses stochastic optimization
to scale up to massive data. We discuss modern research in VI and highlight
important open problems. VI is powerful, but it is not yet well understood.
Our hope in writing this paper is to catalyze statistical research on this
class of algorithms., archivePrefix = arXiv, arxivId = arXiv:1601.00670v7,
author = Blei, David M and Kucukelbir, Alp and Mcauliffe, Jon D, eprint =
arXiv:1601.00670v7, file = ::, keywords = Algorithms,Computationally Intensive
Methods,Statistical Computing, title = Variational Inference: A Review for
Statisticians, url = https://arxiv.org/pdf/1601.00670.pdf, year = 2017
@articleHashim2017, abstract = 2017 Australian and New Zealand Society of
Cardiac and Thoracic Surgeons (ANZSCTS) and the Cardiac Society of Australia
and New Zealand (CSANZ). The revision of an internal mammary artery graft
anastomosis because of a technical error can be time consuming and complicated
and may lead to complications. Here, we describe the technical details and our
early experience of using a standard transit-time flowmeter to exclude
technical errors and facilitate rapid decision making for anastomosis revision
in an arrested heart during aortic cross-clamping in the absence of ultrasound
guidance., author = Hashim, S.A. and Amin, M.A. and Nair, A. and Raja Mokhtar,
R.A. and Krishnasamy, S. and Cheng, K., doi = 10.1016/j.hlc.2017.11.011, issn
= 14442892, journal = Heart Lung and Circulation, keywords = CABG
device,Coronary artery bypass graft, title = A Flowmeter Technique to Exclude
Internal Mammary Artery Anastomosis Error in an Arrested Heart, year = 2017
@articleHestness, abstract = Deep learning (DL) creates impactful advances
following a virtuous recipe: model architecture search, creating large
training data sets, and scaling computation. It is widely believed that
growing training sets and models should improve accuracy and result in better
products. As DL application domains grow, we would like a deeper understanding
of the relationships between training set size, computational scale, and model
accuracy improvements to advance the state-of-the-art. This paper presents a
large scale empirical characterization of generalization error and model size
growth as training sets grow. We introduce a methodology for this measurement
and test four machine learning domains: machine translation, language
modeling, image processing, and speech recognition. Our empirical results show
power-law generalization error scaling across a breadth of factors, resulting
in power-law exponentsâthe ”steepness” of the learning curveâyet to be
explained by theoretical work. Further, model improvements only shift the
error but do not appear to affect the power-law exponent. We also show that
model size scales sublinearly with data size. These scaling relationships have
significant implications on deep learning research, practice, and systems.
They can assist model debugging, setting accuracy targets, and decisions about
data set growth. They can also guide computing system design and underscore
the importance of continued computational scaling., author = Hestness, Joel
and Narang, Sharan and Ardalani, Newsha and Diamos, Gregory and Jun, Heewoo
and Kianinejad, Hassan and Mostofa Ali Patwary, Md and Yang, Yang and Zhou,
Yanqi and Research, Baidu, file = ::, title = Deep Learning Scaling Is
Predictable, Empirically, url = https://arxiv.org/pdf/1712.00409.pdf
@articleLegg2007, abstract = A fundamental problem in artificial intelligence
is that nobody really knows what intelligence is. The problem is especially
acute when we need to consider artificial systems which are significantly
different to humans. In this paper we approach this problem in the following
way: We take a number of well known informal definitions of human intelligence
that have been given by experts, and extract their essential features. These
are then mathematically formalised to produce a general measure of
intelligence for arbitrary machines. We believe that this equation formally
captures the concept of machine intelligence in the broadest reasonable sense.
We then show how this formal definition is related to the theory of universal
optimal learning agents. Finally, we survey the many other tests and
definitions of intelligence that have been proposed for machines.,
archivePrefix = arXiv, arxivId = arXiv:0712.3329v1, author = Legg, Shane and
Hutter, Marcus, eprint = arXiv:0712.3329v1, file = ::, keywords =
AIXI,Complexity theory,Intel-ligence definitions,Intelligence,Intelligence
tests,Theoretical foundations,Turing test, title = Universal Intelligence: A
Definition of Machine Intelligence, url = www.vetta.org/shane www.hutter1.net,
year = 2007 @articleBehrouzi, abstract = Most algorithms for reinforcement
learning work by estimat-ing action-value functions. Here we present a method
that uses Lagrange multipliers, the costate equation, and multi-layer neural
networks to compute policy gradients. We show that this method can find
solutions to time-optimal control problems, driving nonlinear mechanical
systems quickly to a target configuration. On these tasks its performance is
com-parable to that of deep deterministic policy gradient, a recent action-
value method. Research in reinforcement learning has shown the effective-ness
of algorithms based on action-value functions (also known as Q-functions) or
closely related quantities such as value-or advantage functions [1-6]. Here we
present a dif-ferent approach based on Lagrange multipliers and the co-state
equation [7, 8], which may work better in some tasks. As usual in
reinforcement learning, the setting involves an agent embedded in an
environment which evolves through time according to a rule or function f,
called the environment dynamics. For instance, the agent might be a brain and
the environment its body, in which case f might represent the mechanics of
that body. If st is the state of the environment at time t, and at is the
action taken by the agent at this time (say, the motor commands issued by the
brain), then at the next time step, t + ât, the state takes a new value,
author = Behrouzi, Bita and Tweed, Douglas, file = ::, title = Lagrange policy
gradient, url = https://arxiv.org/pdf/1711.05817.pdf @articleKlein, author =
Klein, Dan, file = ::, title = Lagrange Multipliers without Permanent
Scarring, url = https://people.eecs.berkeley.edu/ klein/papers/lagrange-
multipliers.pdf @articleI, abstract = Theories of reward learning in
neuroscience have focused on two families of algorithms, thought to capture
deliberative vs. habitual choice. ” Model-based ” algorithms compute the value
of candidate actions from scratch, whereas ” model-free ” algorithms make
choice more efficient but less flexible by storing pre-computed action values.
We examine an intermediate algorithmic family, the successor representation
(SR), which balances flexibility and efficiency by storing partially computed
action values: predictions about future events. These pre-computation
strategies differ in how they update their choices following changes in a
task. SR’s reliance on stored predictions about future states predicts a
unique signature of insensitivity to changes in the task’s sequence of events,
but flexible adjustment following changes to rewards. We provide evidence for
such differential sensitivity in two behavioral studies with humans. These
results suggest that the SR is a computational substrate for semi-flexible
choice in humans, introducing a subtler, more cognitive notion of habit.,
author = I, Momennejad and Em, Russek and Jh, Cheong and Mm, Botvinick and Nd,
Daw and Sj, Gershman, doi = 10.1101/083824, file = ::, keywords = Decision
making,Human behavior,Model-based,Planning,Predictive representation
Acknowledgements,Reinforcement learning,Retrospective revaluation,Successor
representation, title = The successor representation in human reinforcement
learning, url =
https://www.biorxiv.org/content/biorxiv/early/2017/07/04/083824.full.pdf
@articleDayan, abstract = Estimation of returns over time, the focus of
temporal difference (TD) algorithms, imposes particular constraints on good
function approximators or representations. Appropriate generalisation between
states is determined by how similar their succes-sors are, and representations
should follow suit. This paper shows how TD machinery can be used to learn
such representations, and illustrates, using a navigation task, the
appropriately distributed nature of the result., author = Dayan, Peter, file =
::, title = Improving Generalisation for Temporal Difference Learning: The
Successor Representation, url = http://www.gatsby.ucl.ac.uk/
dayan/papers/d93b.pdf @articleNorouzi2017, abstract = A key problem in
structured output prediction is direct optimization of the task reward
function that matters for test evaluation. This paper presents a simple and
computationally efficient approach to incorporate task reward into a maximum
likelihood framework. By establishing a link between the log-likelihood and
ex-pected reward objectives, we show that an optimal regularized expected
reward is achieved when the conditional distribution of the outputs given the
inputs is pro-portional to their exponentiated scaled rewards. Accordingly, we
present a frame-work to smooth the predictive probability of the outputs using
their corresponding rewards. We optimize the conditional log-probability of
augmented outputs that are sampled proportionally to their exponentiated
scaled rewards. Experiments on neural sequence to sequence models for speech
recognition and machine transla-tion show notable improvements over a maximum
likelihood baseline by using reward augmented maximum likelihood (RAML), where
the rewards are defined as the negative edit distance between the outputs and
the ground truth labels., archivePrefix = arXiv, arxivId = arXiv:1609.00150v3,
author = Norouzi, Mohammad and Bengio, Samy and Chen, Zhifeng and Jaitly,
Navdeep and Schuster, Mike and Wu, Yonghui and Schuurmans, Dale, eprint =
arXiv:1609.00150v3, file = ::, keywords = (), title = Reward Augmented Maximum
Likelihood for Neural Structured Prediction, url =
https://arxiv.org/pdf/1609.00150.pdf, year = 2017 @articlePetersa, abstract =
Many robot control problems of practical im-portance, including operational
space con-trol, can be reformulated as immediate re-ward reinforcement
learning problems. How-ever, few of the known optimization or re-inforcement
learning algorithms can be used in online learning control for robots, as they
are either prohibitively slow, do not scale to interesting domains of complex
robots, or re-quire trying out policies generated by random search, which are
infeasible for a physical sys-tem. Using a generalization of the EM-base
reinforcement learning framework suggested by Dayan & Hinton, we reduce the
prob-lem of learning with immediate rewards to a reward-weighted regression
problem with an adaptive, integrated reward transforma-tion for faster
convergence. The resulting al-gorithm is efficient, learns smoothly without
dangerous jumps in solution space, and works well in applications of complex
high degree-of-freedom robots., author = Peters, Jan and Schaal, Stefan, file
= ::, title = Reinforcement Learning by Reward-weighted Regression for
Operational Space Control, url =
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.79.6266&rep=rep1&type=pdf
@articleLecun2006, abstract = To appear in ” Predicting Structured Data ” , G.
Bakir, T. Hofman, B. Schölkopf, A. Smola, B. Taskar (eds) MIT Press, 2006
Abstract Energy-Based Models (EBMs) capture dependencies between variables by
as-sociating a scalar energy to each configuration of the variables. Inference
consists in clamping the value of observed variables and finding
configurations of the re-maining variables that minimize the energy. Learning
consists in finding an energy function in which observed configurations of the
variables are given lower energies than unobserved ones. The EBM approach
provides a common theoretical frame-work for many learning models, including
traditional discriminative and genera-tive approaches, as well as graph-
transformer networks, conditional random fields, maximum margin Markov
networks, and several manifold learning methods. Probabilistic models must be
properly normalized, which sometimes requires evaluating intractable integrals
over the space of all possible variable configura-tions. Since EBMs have no
requirement for proper normalization, this problem is naturally circumvented.
EBMs can be viewed as a form of non-probabilistic factor graphs, and they
provide considerably more flexibility in the design of architec-tures and
training criteria than probabilistic approaches., author = Lecun, Yann and
Chopra, Sumit and Hadsell, Raia and Ranzato, Marc Aurelio and Huang, Fu Jie,
file = ::, title = A Tutorial on Energy-Based Learning, url =
http://yann.lecun.com, year = 2006 @articleAnderson1999, abstract = History
and definition: The term ” Monte Carlo ” was apparently first used by Ulam and
von Neumann as a Los Alamos code word for the stochastic simulations they
applied to building better atomic bombs. Their methods, involving the laws of
chance, were aptly named after the inter-national gaming destination; the
moniker stuck and soon after the War a wide range of sticky problems yielded
to the new techniques. Despite the widespread use of the methods, and numerous
descriptions of them in articles and monographs, it is virtually impossible to
find a succint defini-tion of ” Monte Carlo method ” in the literature.
Perhaps this is owing to the intuitive nature of the topic which spawns many
definitions by way of specific examples. Some authors prefer to use the term ”
stochastic simulation ” for almost everything, reserving ” Monte Carlo ” only
for Monte Carlo Integration and Monte Carlo Tests (cf. Ripley 1987). Others
seem less concerned about blurring the distinction between simulation studies
and Monte Carlo methods. Be that as it may, a suitable definition can be good
to have, if for nothing other than to avoid the awkwardness of trying to
define the Monte Carlo method by appealing to a whole bevy of examples of it.
Since I am (so Elizabeth claims!) unduly influenced by my advisor’s ways of
thinking, I like to define Monte Carlo in the spirit of definitions she has
used before. In particular, I use: Definition: Monte Carlo is the art of
approximating an expectation by the sample mean of a function of simulated
random variables. We will find that this definition is broad enough to cover
everything that has been called Monte Carlo, and yet makes clear its essence
in very familiar terms: Monte Carlo is about invoking laws of large numbers to
approximate expectations. 1 While most Monte Carlo simulations are done by
computer today, there were many applications of Monte Carlo methods using
coin-flipping, card-drawing, or needle-tossing (rather than computer-generated
pseudo-random numbers) as early as the turn of the centuryâlong before the
name Monte Carlo arose. In more mathematical terms: Consider a (possibly
multidimensional) random variable X having probability mass function or
probability density function f X (x) which is greater than zero on a set of
values X . Then the expected value of a function g of X is, author = Anderson,
Eric C, file = ::, title = Lecture Notes for Stat 578C Monte Carlo Methods and
Importance Sampling, url =
http://ib.berkeley.edu/labs/slatkin/eriq/classes/guest_lect/mc_lecture_notes.pdf,
year = 1999 @articleBengioa, abstract = Neuroscientists have long criticised
deep learn-ing algorithms as incompatible with current knowledge of
neurobiology. We explore more bi-ologically plausible versions of deep
representa-tion learning, focusing here mostly on unsuper-vised learning but
developing a learning mecha-nism that could account for supervised, unsuper-
vised and reinforcement learning. The starting point is that the basic
learning rule believed to govern synaptic weight updates (Spike-Timing-
Dependent Plasticity) arises out of a simple up-date rule that makes a lot of
sense from a machine learning point of view and can be interpreted as gradient
descent on some objective function so long as the neuronal dynamics push
firing rates towards better values of the objective func-tion (be it
supervised, unsupervised, or reward-driven). The second main idea is that this
corre-sponds to a form of the variational EM algorithm, i.e., with approximate
rather than exact posteri-ors, implemented by neural dynamics. Another
contribution of this paper is that the gradients re-quired for updating the
hidden states in the above variational interpretation can be estimated using
an approximation that only requires propagating activations forward and
backward, with pairs of layers learning to form a denoising auto-encoder.
Finally, we extend the theory about the proba-bilistic interpretation of auto-
encoders to justify improved sampling schemes based on the gener-ative
interpretation of denoising auto-encoders, and we validate all these ideas on
generative learning tasks., author = Bengio, Yoshua and Lee, Dong-Hyun and
Bornschein, Jorg and Mesnard, Thomas and Lin, Zhouhan, file = ::, title =
Towards Biologically Plausible Deep Learning, url =
https://arxiv.org/pdf/1502.04156.pdf @articleRivest, abstract = Successful
application of reinforcement learning algorithms often involves considerable
hand-crafting of the necessary non-linear features to reduce the complexity of
the value functions and hence to promote convergence of the algorithm. In
contrast, the human brain readily and autonomously finds the complex features
when provided with sufficient training. Recent work in machine learning and
neurophysiology has demonstrated the role of the basal ganglia and the frontal
cortex in mammalian reinforcement learning. This paper develops and explores
new reinforcement learning algorithms inspired by neurological evidence that
provides potential new approaches to the feature construction problem. The
algorithms are compared and evaluated on the Acrobot task., author = Rivest,
François and Bengio, Yoshua and Kalaska, John, file = ::, title = Brain
Inspired Reinforcement Learning, url =
https://papers.nips.cc/paper/2749-brain-inspired-reinforcement-learning.pdf
@inproceedingsDosovitskiy, abstract = We present an approach to sensorimotor
control in immersive environments. Our approach utilizes a high-dimensional
sensory stream and a lower-dimensional measurement stream. The cotemporal
structure of these streams provides a rich supervisory signal, which enables
training a sensorimotor control model by in-teracting with the environment.
The model is trained using supervised learning techniques, but without
extraneous supervision. It learns to act based on raw sen-sory input from a
complex three-dimensional environment. The presented formu-lation enables
learning without a fixed goal at training time, and pursuing dynam-ically
changing goals at test time. We conduct extensive experiments in three-
dimensional simulations based on the classical first-person game Doom. The
results demonstrate that the presented approach outperforms sophisticated
prior formulations, particularly on challenging tasks. The results also show
that trained models successfully generalize across environments and goals. A
model trained using the presented approach won the Full Deathmatch track of
the Visual Doom AI Competition, which was held in previously unseen
environments., author = Dosovitskiy, Alexey and Koltun, Vladlen, booktitle =
International Conference on Learning Representations (ICLR), file = ::, title
= Learning To Act By Predicting The Future, url =
https://arxiv.org/pdf/1611.01779.pdf, year = 2017 @articleSilvera, author =
Silver, David, file = ::, title = Lecture 7: Policy Gradient, url =
http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/pg.pdf
@articleLevine, author = Levine, Sergey, file = ::, title = Connections
Between Inference and Control CS 294-112: Deep Reinforcement Learning, url =
http://rll.berkeley.edu/deeprlcourse/f17docs/lecture_11_control_and_inference.pdf
@articleZiebart, abstract = The principle of maximum entropy provides a
powerful framework for statistical models of joint, conditional, and marginal
distribu-tions. However, there are many important distributions with elements
of interaction and feedback where its applicability has not been established.
This work presents the principle of maximum causal entropyâan approach based
on causally conditioned probabilities that can appropriately model the
availabil-ity and influence of sequentially revealed side information. Using
this principle, we de-rive models for sequential data with revealed
information, interaction, and feedback, and demonstrate their applicability
for statisti-cally framing inverse optimal control and de-cision prediction
tasks., author = Ziebart, Brian D and Bagnell, J Andrew and Dey, Anind K, file
= ::, title = Modeling Interaction via the Principle of Maximum Causal
Entropy, url = http://www.cs.cmu.edu/ bziebart/publications/maximum-causal-
entropy.pdf @inproceedingsbojanowski2017unsupervised, abstract = Convolutional
neural networks provide visual features that perform remarkably well in many
computer vision applications. However, train-ing these networks requires
significant amounts of supervision. This paper introduces a generic framework
to train deep networks, end-to-end, with no supervision. We propose to fix a
set of target representations, called Noise As Targets (NAT), and to constrain
the deep features to align to them. This domain agnostic approach avoids the
standard unsupervised learning issues of triv-ial solutions and collapsing of
features. Thanks to a stochastic batch reassignment strategy and a separable
square loss function, it scales to mil-lions of images. The proposed approach
pro-duces representations that perform on par with state-of-the-art
unsupervised methods on Ima-geNet and PASCAL VOC., author = Bojanowski, Piotr
and Joulin, Armand, booktitle = International Conference on Machine Learning
(ICML), file = ::, title = Unsupervised Learning by Predicting Noise, url =
https://arxiv.org/pdf/1704.05310.pdf, year = 2017 @articleRolf, abstract = We
discuss the difficulty to learn control skills in high-dimensional domains
that can not be exhaustively explored. We show how infant-development can
serve as a role-model for highly efficient exploration: infants neither
explore exhaustively, nor do they learn very versatile skills right in the
beginning. Rather, they attempt goal-directed exploration to achieve
feedforward control as a first step without re-quiring full knowledge of the
world. This article reviews recent efforts to mimick such pathways by means of
” goal babbling ” , which have led to a series of algo-rithms that allow for a
likewise efficient learning. We show that it permits to learn inverse models
from examples even in the presence of non-convex solution sets by utilizing a
reward-weighted regression scheme, and that a human-competitive learning speed
can be achieved if online learning is applied ” in the loop ” . Results are
verified on the ” Bionic Handling Assistant ” , a novel bionic robot that
instan-tiates a wide spread of problems like high dimensions, non-stationary
behavior, highly constrained actuators, sensory noise, and very slow response-
behavior., author = Rolf, Matthias and Asada, Minoru, file = ::, title =
Learning Inverse Models in High Dimensions with Goal Babbling and Reward-
Weighted Averaging, url =
http://acl.mit.edu/amlsc/files/amlsc13_submission_7.pdf @articleKremer,
abstract = In machine learning, active learning refers to algorithms that
autonomously select the data points from which they will learn. There are many
data mining appli-cations in which large amounts of unlabeled data are readily
available, but labels (e.g., human annotations or results from complex
experiments) are costly to ob-tain. In such scenarios, an active learning
algorithm aims at identifying data points that, if labeled and used for
training, would most improve the learned model. La-bels are then obtained only
for the most promising data points. This speeds up learning and reduces
labeling costs. Support vector machine (SVM) classifiers are particularly
well-suited for active learning due to their convenient mathemat-ical
properties. They perform linear classification, typically in a kernel-induced
feature space, which makes measuring the distance of a data point from the de-
cision boundary straightforward. Furthermore, heuristics can efficiently
estimate how strongly learning from a data point influences the current model.
This infor-mation can be used to actively select training samples. After a
brief introduction to the active learning problem, we discuss different query
strategies for select-ing informative data points and review how these
strategies give rise to different variants of active learning with SVMs.,
author = Kremer, Jan and Pedersen, Kim Steenstrup and Igel, Christian, file =
::, title = Active Learning with Support Vector Machines, url =
http://image.diku.dk/jank/papers/WIREs2014.pdf @articleJonas2017, author =
Jonas, Eric and Kording, Konrad Paul, doi = 10.1371/journal.pcbi.1005268,
editor = Diedrichsen, Jörn, file = ::, issn = 1553-7358, journal = PLOS
Computational Biology, month = jan, number = 1, pages = e1005268, publisher =
Public Library of Science, title = Could a Neuroscientist Understand a
Microprocessor?, url = http://dx.plos.org/10.1371/journal.pcbi.1005268, volume
= 13, year = 2017 @articleCzarnecki, abstract = At the heart of deep learning
we aim to use neural networks as function approxi-mators â training them to
produce outputs from inputs in emulation of a ground truth function or data
creation process. In many cases we only have access to input-output pairs from
the ground truth, however it is becoming more common to have access to
derivatives of the target output with respect to the input â for exam-ple
when the ground truth function is itself a neural network such as in network
compression or distillation. Generally these target derivatives are not
computed, or are ignored. This paper introduces Sobolev Training for neural
networks, which is a method for incorporating these target derivatives in
addition the to target values while training. By optimising neural networks to
not only approximate the func-tion’s outputs but also the function’s
derivatives we encode additional information about the target function within
the parameters of the neural network. Thereby we can improve the quality of
our predictors, as well as the data-efficiency and generalization capabilities
of our learned function approximation. We provide theoretical justifications
for such an approach as well as examples of empirical evidence on three
distinct domains: regression on classical optimisation datasets, distilling
policies of an agent playing Atari, and on large-scale applications of
synthetic gradients. In all three domains the use of Sobolev Training,
employing target derivatives in addition to target values, results in models
with higher accuracy and stronger generalisation., author = Czarnecki,
Wojciech Marian and Osindero, Simon and Jaderberg, Max and Swirszcz, Grzegorz
and Pascanu, Razvan, file = ::, title = Sobolev Training for Neural Networks,
url = https://arxiv.org/pdf/1706.04859.pdf @articleOdonoghue2017, abstract =
We consider the exploration/exploitation problem in reinforcement learning.
For exploitation, it is well known that the Bellman equation connects the
value at any time-step to the expected value at subsequent time-steps. In this
paper we consider a similar uncertainty Bellman equation (UBE), which connects
the uncertainty at any time-step to the expected uncertainties at subsequent
time-steps, thereby extending the potential exploratory benefit of a policy
beyond individual time-steps. We prove that the unique fixed point of the UBE
yields an upper bound on the variance of the estimated value of any fixed
policy. This bound can be much tighter than traditional count-based bonuses
that compound standard deviation rather than variance. Importantly, and unlike
several existing approaches to optimism, this method scales naturally to large
systems with complex generalization. Substituting our UBE-exploration strategy
for -greedy improves DQN performance on 51 out of 57 games in the Atari
suite., author = O ’donoghue, Brendan and Osband, Ian and Munos, Remi and
Deepmind, Volodymyr Mnih, file = ::, title = The Uncertainty Bellman Equation
and Exploration, url = https://arxiv.org/pdf/1709.05380.pdf, year = 2017
@articleVasilaki2017, abstract = The Epicurean Philosophy is commonly thought
as sim-plistic and hedonistic. Here I discuss how this is a misconception and
explore its link to Reinforcement Learning. Based on the letters of Epicurus,
I construct an objective function for hedonism which turns out to be
equivalent of the Reinforcement Learning objective function when omitting the
discount factor. I then dis-cuss how Plato and Aristotle ’s views that can be
also loosely linked to Reinforcement Learning, as well as their weaknesses in
relationship to it. Finally, I em-phasise the close affinity of the Epicurean
views and the Bellman equation., archivePrefix = arXiv, arxivId =
arXiv:1710.04582v1, author = Vasilaki, Eleni, eprint = arXiv:1710.04582v1,
file = ::, title = Is Epicurus the father of Reinforcement Learning?, url =
https://arxiv.org/pdf/1710.04582.pdf, year = 2017
@inproceedingsosband2016bootstrapped, abstract = Efficient exploration remains
a major challenge for reinforcement learning (RL). Common dithering strategies
for exploration, such as -greedy, do not carry out temporally-extended (or
deep) exploration; this can lead to exponentially larger data requirements.
However, most algorithms for statistically efficient RL are not
computationally tractable in complex en-vironments. Randomized value functions
offer a promising approach to efficient exploration with generalization, but
existing algorithms are not compatible with nonlinearly parameterized value
functions. As a first step towards addressing such contexts we develop
bootstrapped DQN. We demon-strate that bootstrapped DQN can combine deep
exploration with deep neural networks for exponentially faster learning than
any dithering strat-egy. In the Arcade Learning Environment bootstrapped DQN
substantially improves learning speed and cumulative performance across most
games., author = Osband, Ian and Blundell, Charles and Pritzel, Alexander and
Roy, Benjamin Van, booktitle = Advances in Neural Information Processing
Systems (NIPS), file = ::, title = Deep Exploration via Bootstrapped DQN, url
= https://arxiv.org/pdf/1602.04621.pdf, year = 2016 @articleDeepmind, abstract
= The deep reinforcement learning community has made sev-eral independent
improvements to the DQN algorithm. How-ever, it is unclear which of these
extensions are complemen-tary and can be fruitfully combined. This paper
examines six extensions to the DQN algorithm and empirically studies their
combination. Our experiments show that the combina-tion provides state-of-the-
art performance on the Atari 2600 benchmark, both in terms of data efficiency
and final perfor-mance. We also provide results from a detailed ablation study
that shows the contribution of each component to overall per-formance., author
= Deepmind, Matteo Hessel and Deepmind, Joseph Modayil and Van, Hado and
Deepmind, Hasselt and Deepmind, Tom Schaul and Ostrovski, Georg and Will,
Deepmind and Deepmind, Dabney and Horgan, Dan and Bilal, Deepmind and
Deepmind, Piot and Azar, Mohammad and David, Deepmind and Deepmind, Silver,
file = ::, title = Rainbow: Combining Improvements in Deep Reinforcement
Learning, url = https://arxiv.org/pdf/1710.02298.pdf
@inproceedingsbyravan2018se3pose, abstract = â In this work, we present an
approach to deep visuomotor control using structured deep dynamics models. Our
deep dynamics model, a variant of SE3-Nets, learns a low-dimensional pose
embedding for visuomotor control via an encoder-decoder structure. Unlike
prior work, our dynamics model is structured: given an input scene, our
network ex-plicitly learns to segment salient parts and predict their pose-
embedding along with their motion modeled as a change in the pose space due to
the applied actions. We train our model using a pair of point clouds separated
by an action and show that given supervision only in the form of point-wise
data associations between the frames our network is able to learn a meaningful
segmentation of the scene along with consistent poses. We further show that
our model can be used for closed-loop control directly in the learned low-
dimensional pose space, where the actions are computed by minimizing error in
the pose space using gradient-based methods, similar to traditional model-
based control. We present results on controlling a Baxter robot from raw depth
data in simulation and in the real world and compare against two baseline deep
networks. Our method runs in real-time, achieves good prediction of scene
dynamics and outperforms the baseline methods on multiple control runs. Video
results can be found at: https://rse-lab.cs. washington.edu/se3-structured-
deep-ctrl/, author = Byravan, Arunkumar and Leeb, Felix and Meier, Franziska
and Fox, Dieter, booktitle = IEEE International Conference on Robotics and
Automation (ICRA), file = ::, title = SE3-Pose-Nets: Structured Deep Dynamics
Models for Visuomotor Planning and Control, url =
https://arxiv.org/pdf/1710.00489.pdf, year = 2018 @articlePritzel, abstract =
Deep reinforcement learning methods attain super-human performance in a wide
range of en-vironments. Such methods are grossly inefficient, often taking
orders of magnitudes more data than humans to achieve reasonable performance.
We propose Neural Episodic Control: a deep rein-forcement learning agent that
is able to rapidly assimilate new experiences and act upon them. Our agent
uses a semi-tabular representation of the value function: a buffer of past
experience con-taining slowly changing state representations and rapidly
updated estimates of the value function. We show across a wide range of
environments that our agent learns significantly faster than other state-of-
the-art, general purpose deep reinforce-ment learning agents., author =
Pritzel, Alexander and Uria, Benigno and Srinivasan, Sriram and Com,
Adriap@google and Vinyals, Oriol and Hassabis, Demis and Wierstra, Daan and
Blundell, Charles, file = ::, title = Neural Episodic Control Adrì aPuigdo
enech, url = https://arxiv.org/pdf/1703.01988.pdf @articleVenkatraman,
abstract = Recurrent neural networks (RNNs) are a vital modeling technique
that rely on internal states learned indirectly by optimization of a
supervised, unsupervised, or reinforcement training loss. RNNs are used to
model dynamic processes that are characterized by underlying latent states
whose form is often unknown, precluding its analytic representation inside an
RNN. In the Predictive-State Representation (PSR) literature, latent state
processes are modeled by an internal state representa-tion that directly
models the distribution of future observations, and most recent work in this
area has relied on explicitly representing and targeting sufficient statis-
tics of this probability distribution. We seek to combine the advantages of
RNNs and PSRs by augmenting existing state-of-the-art recurrent neural
networks with PREDICTIVE-STATE DECODERS (PSDs), which add supervision to the
network’s internal state representation to target predicting future
observations. PSDs are simple to implement and easily incorporated into
existing training pipelines via additional loss regularization. We demonstrate
the effectiveness of PSDs with experimental results in three different
domains: probabilistic filtering, Imitation Learning, and Reinforcement
Learning. In each, our method improves statistical performance of state-of-
the-art recurrent baselines and does so with fewer iterations and less data.,
author = Venkatraman, Arun and Rhinehart, Nicholas and Sun, Wen and Pinto,
Lerrel and Hebert, Martial and Boots, Byron and Kitani, Kris M and Bagnell, J
Andrew, file = ::, title = Predictive-State Decoders: Encoding the Future into
Recurrent Networks, url = https://arxiv.org/pdf/1709.08520.pdf
@inproceedingskaiser2017rareevents, author = Kaiser, Lukasz and Nachum, Ofir
and Aurko, Roy and Bengio, Samy, booktitle = International Conference on
Learning Representations (ICLR), file = ::, title = Learning to Remember Rare
Events, year = 2017 @articleGhahramani2015, abstract = T he key idea behind
the probabilistic framework to machine learn-ing is that learning can be
thought of as inferring plausible models to explain observed data. A machine
can use such models to make predictions about future data, and take decisions
that are rational given these predictions. Uncertainty plays a fundamental
part in all of this. Observed data can be consistent with many models, and
therefore which model is appropriate, given the data, is uncertain. Similarly,
predictions about future data and the future consequences of actions are
uncertain. Probability theory provides a framework for modelling uncertainty.
This Review starts with an introduction to the probabilistic approach to
machine learning and Bayesian inference, and then discusses some of the state-
of-the-art advances in the field. Many aspects of learning and intelligence
crucially depend on the careful probabilistic representation of uncertainty.
Probabilistic approaches have only recently become a main-stream approach to
artificial intelligence 1 , robotics 2 and machine learn-ing 3,4 . Even now,
there is controversy in these fields about how important it is to fully
represent uncertainty. For example, advances using deep neural networks to
solve challenging pattern-recognition problems such as speech recognition 5 ,
image classification 6,7 , and prediction of words in text 8 , do not overtly
represent the uncertainty in the structure or parameters of those neural
networks. However, my focus will not be on these types of pattern-recognition
problems, characterized by the availability of large amounts of data, but on
problems for which uncertainty is really a key ingredient, for example where a
decision may depend on the amount of uncertainty. I highlight five areas of
current research at the frontier of probabilistic machine learning,
emphasizing areas that are of broad relevance to sci-entists across many
fields: probabilistic programming, which is a general framework for expressing
probabilistic models as computer programs and which could have a major impact
on scientific modelling; Bayes-ian optimization, which is an approach to
globally optimizing unknown functions; probabilistic data compression;
automating the discovery of plausible and interpretable models from data; and
hierarchical modelling for learning many related models, for example for
personalized medicine or recommendation. Although considerable challenges
remain, the com-ing decade promises substantial advances in artificial
intelligence and machine learning based on the probabilistic framework.,
author = Ghahramani, Zoubin, doi = 10.1038/nature14541, file = ::, journal =
Nature, title = Probabilistic modelling and representing uncertainty, url =
https://www.cse.iitk.ac.in/users/piyush/courses/pml_winter16/nature14541.pdf,
volume = 521, year = 2015 @articleBrooks1991, abstract = Artificial
intelligence research has foundered on the issue of representation. When
intelligence is approached in an incremental manner, with strict reliance on
interfacing to the real world through perception and action, reliance on
representation disappears. In this paper we outline our approach to
incrementally building complete intelligent Creatures. The fundamental
decomposition of the intelligent system is not into independent information
processing units which must interface with each other via representations.
Instead, the intelligent system is decomposed into independent and parallel
activity producers which all interface directly to the world through
perception and action, rather than interface to each other particularly much.
The notions of central and peripheral systems evaporateeverything is both
central and peripheral. Based on these principles we have built a very
successful series of mobile robots which operate without supervision as
Creatures in standard office environments., author = Brooks, Rodney A, file =
::, journal = Artificial Intelligence, pages = 139–159, title = Intelligence
without representation*, url =
http://people.csail.mit.edu/brooks/papers/representation.pdf, volume = 47,
year = 1991 @inproceedingsflorensa2018automatic, title=Automatic goal
generation for reinforcement learning agents, author=Florensa, Carlos and
Held, David and Geng, Xinyang and Abbeel, Pieter, booktitle=International
conference on machine learning, pages=1515–1528, year=2018
@inproceedingsschulman2016gae, abstract = Policy gradient methods are an
appealing approach in reinforcement learning be-cause they directly optimize
the cumulative reward and can straightforwardly be used with nonlinear
function approximators such as neural networks. The two main challenges are
the large number of samples typically required, and the diffi-culty of
obtaining stable and steady improvement despite the nonstationarity of the
incoming data. We address the first challenge by using value functions to
substan-tially reduce the variance of policy gradient estimates at the cost of
some bias, with an exponentially-weighted estimator of the advantage function
that is analogous to TD($\lambda$). We address the second challenge by using
trust region optimization procedure for both the policy and the value
function, which are represented by neural networks. Our approach yields strong
empirical results on highly challenging 3D locomo-tion tasks, learning running
gaits for bipedal and quadrupedal simulated robots, and learning a policy for
getting the biped to stand up from starting out lying on the ground. In
contrast to a body of prior work that uses hand-crafted policy repre-
sentations, our neural network policies map directly from raw kinematics to
joint torques. Our algorithm is fully model-free, and the amount of simulated
experi-ence required for the learning tasks on 3D bipeds corresponds to 1-2
weeks of real time., author = Schulman, John and Moritz, Philipp and Levine,
Sergey and Jordan, Michael I and Abbeel, Pieter, booktitle = International
Conference on Learning Representations (ICLR), file = ::, title = High-
Dimensional Continuous Control Using Generalized Advantage Estimation, url =
https://arxiv.org/pdf/1506.02438.pdf, year = 2016 @articlePastor2011, abstract
= on PR2, author = Pastor, Peter, doi = 10.1109/ICRA.2011.5980200, file = ::,
isbn = 978-1-61284-386-5, issn = 1050-4729, journal = IEEE International
Conference on Robotics and Automation (ICRA), pages = 3828–3834, title = Skill
Learning and Task Outcome Prediction for Manipulation, url =
http://www.cs.utexas.edu/ sniekum/classes/RLFD-F16/papers/Pastor11.pdf, year =
2011 @articleGhadirzadeh, abstract = â Skilled robot task learning is best
implemented by predictive action policies due to the inherent latency of
sensorimotor processes. However, training such predictive poli-cies is
challenging as it involves finding a trajectory of motor activations for the
full duration of the action. We propose a data-efficient deep predictive
policy training (DPPT) framework with a deep neural network policy
architecture which maps an image observation to a sequence of motor
activations. The architecture consists of three sub-networks referred to as
the perception, policy and behavior super-layers. The perception and behavior
super-layers force an abstraction of visual and motor data trained with
synthetic and simulated training samples, respectively. The policy super-layer
is a small sub-network with fewer parameters that maps data in-between the
abstracted manifolds. It is trained for each task using methods for policy
search reinforcement learning. We demonstrate the suitability of the proposed
architecture and learning framework by training predictive policies for
skilled object grasping and ball throwing on a PR2 robot. The effectiveness of
the method is illustrated by the fact that these tasks are trained using only
about 180 real robot attempts with qualitative terminal rewards., author =
Ghadirzadeh, Ali and Maki, Atsuto and Kragic, Danica and Björkman, Mårten,
file = ::, title = Deep Predictive Policy Training using Reinforcement
Learning, url = https://arxiv.org/pdf/1703.00727.pdf @articlePlappert,
abstract = Deep reinforcement learning (RL) methods generally engage in
exploratory be-havior through noise injection in the action space. An
alternative is to add noise directly to the agent’s parameters, which can lead
to more consistent exploration and a richer set of behaviors. Methods such as
evolutionary strategies use parameter perturbations, but discard all temporal
structure in the process and require signif-icantly more samples. Combining
parameter noise with traditional RL methods allows to combine the best of both
worlds. We demonstrate that both off-and on-policy methods benefit from this
approach through experimental comparison of DQN, DDPG, and TRPO on high-
dimensional discrete action environments as well as continuous control tasks.
Our results show that RL with parameter noise learns more efficiently than
traditional RL with action space noise and evolutionary strategies
individually., author = Plappert, Matthias and Houthooft, Rein and Dhariwal,
Prafulla and Sidor, Szymon and Chen, Richard Y and Chen, Xi and Asfour, Tamim
and Abbeel, Pieter and Openai, Marcin Andrychowicz, file = ::, title =
Parameter Space Noise for Exploration, url =
https://arxiv.org/pdf/1706.01905.pdf @articleRajeswaran, abstract = Sample
complexity and safety are major challenges when learning policies with
reinforcement learning for real-world tasks, especially when the policies are
repre-sented using rich function approximators like deep neural networks.
Model-based methods where the real-world target domain is approximated using a
simulated source domain provide an avenue to tackle the above challenges by
augmenting real data with simulated data. However, discrepancies between the
simulated source domain and the target domain pose a challenge for simulated
training. We introduce the EPOpt algorithm, which uses an ensemble of
simulated source domains and a form of adversarial training to learn policies
that are robust and generalize to a broad range of possible target domains,
including unmodeled effects. Further, the probability distribution over source
domains in the ensemble can be adapted using data from target domain and
approximate Bayesian methods, to progressively make it a better approximation.
Thus, learning on a model ensemble, along with source domain adaptation,
provides the benefit of both robustness and learning/adaptation., author =
Rajeswaran, Aravind and Ghotra, Sarvjeet and Ravindran, Balaraman and Levine,
Sergey, file = ::, title = EPOPT: LEARNING ROBUST NEURAL NETWORK POLICIES
USING MODEL ENSEMBLES, url = https://arxiv.org/pdf/1610.01283.pdf
@inproceedingsBansal, abstract = â Real-world robots are becoming
increasingly com-plex and commonly act in poorly understood environments where
it is extremely challenging to model or learn their true dynamics. Therefore,
it might be desirable to take a task-specific approach, wherein the focus is
on explicitly learning the dynamics model which achieves the best control
performance for the task at hand, rather than learning the true dynamics. In
this work, we use Bayesian optimization in an active learning framework where
a locally linear dynamics model is learned with the intent of maximizing the
control performance, and used in conjunction with optimal control schemes to
efficiently design a controller for a given task. This model is updated
directly based on the performance observed in experiments on the physical
system in an iterative manner until a desired performance is achieved. We
demonstrate the efficacy of the proposed approach through simulations and real
experiments on a quadrotor testbed., author = Bansal, Somil and Calandra,
Roberto and Xiao, Ted and Levine, Sergey and Tomlin, Claire J, booktitle =
IEEE Conference on Decision and Control (CDC), file = ::, title = Goal-Driven
Dynamics Learning via Bayesian Optimization, url =
https://arxiv.org/pdf/1703.09260.pdf, year = 2017 @articleCalandra2017, author
= Calandra, Roberto, file = ::, title = Goal-Driven Dynamics Learning for
Model-Based RL Motivation, url = http://www.robertocalandra.com/wp-
content/uploads/2017-04-20-dali.pdf, year = 2017 @inproceedingsTamar2016,
abstract = We introduce the value iteration network: a fully differentiable
neural network with a ‘planning module’ embedded within. Value iteration
networks are suitable for making predictions about outcomes that involve
planning-based reasoning, such as predicting a desired trajectory from an
observation of a map. Key to our approach is a novel differentiable
approximation of the value-iteration algorithm, which can be represented as a
convolutional neural network, and trained end-to-end using standard
backpropagation. We evaluate our value iteration networks on the task of
predicting optimal obstacle-avoiding trajectories from an image of a
landscape, both on synthetic data, and on challenging raw images of the Mars
terrain., archivePrefix = arXiv, arxivId = 1602.02867, author = Tamar, Aviv
and Levine, Sergey and Abbeel, Pieter, booktitle = Advances in Neural
Information Processing Systems (NIPS), eprint = 1602.02867, issn = 10495258,
pmid = 172808, title = Value Iteration Networks, year = 2016
@inproceedingshouthooft2016vime, abstract = Scalable and effective exploration
remains a key challenge in reinforcement learn-ing (RL). While there are
methods with optimality guarantees in the setting of dis-crete state and
action spaces, these methods cannot be applied in high-dimensional deep RL
scenarios. As such, most contemporary RL relies on simple heuristics such as
exploration or adding Gaussian noise to the controls. This paper introduces
Variational Information Maximizing Exploration (VIME), an explo-ration
strategy based on maximization of information gain about the agent’s belief of
environment dynamics. We propose a practical implementation, using varia-
tional inference in Bayesian neural networks which efficiently handles
continuous state and action spaces. VIME modifies the MDP reward function, and
can be applied with several different underlying RL algorithms. We demonstrate
that VIME achieves significantly better performance compared to heuristic
exploration methods across a variety of continuous control tasks and
algorithms, including tasks with very sparse rewards., archivePrefix = arXiv,
arxivId = 1605.09674, author = Houthooft, Rein and Chen, Xi and Duan, Yan and
Schulman, John and De Turck, Filip and Abbeel, Pieter, booktitle = Advances in
Neural Information Processing Systems (NIPS), eprint = 1605.09674, issn =
10495258, title = Variational Information Maximizing Exploration, year = 2016
@articleReed, abstract = We propose the neural programmer-interpreter (NPI): a
recurrent and composi-tional neural network that learns to represent and
execute programs. NPI has three learnable components: a task-agnostic
recurrent core, a persistent key-value pro-gram memory, and domain-specific
encoders that enable a single NPI to operate in multiple perceptually diverse
environments with distinct affordances. By learning to compose lower-level
programs to express higher-level programs, NPI reduces sample complexity and
increases generalization ability compared to sequence-to-sequence LSTMs. The
program memory allows efficient learning of additional tasks by building on
existing programs. NPI can also harness the environment (e.g. a scratch pad
with read-write pointers) to cache intermediate results of com-putation,
lessening the long-term memory burden on recurrent hidden units. In this work
we train the NPI with fully-supervised execution traces; each program has
example sequences of calls to the immediate subprograms conditioned on the
input. Rather than training on a huge number of relatively weak labels, NPI
learns from a small number of rich examples. We demonstrate the capability of
our model to learn several types of compositional programs: addition, sorting,
and canonicalizing 3D models. Furthermore, a single NPI learns to execute
these pro-grams and all 21 associated subprograms., author = Reed, Scott and
De Freitas, Nando, file = ::, title = Neural Programmer-Interpreters, url =
https://arxiv.org/pdf/1511.06279v3.pdf @inproceedingsnagaband2018mbmf,
abstract = Model-free deep reinforcement learning methods have successfully
learned com-plex behavioral strategies for a wide range of tasks, but
typically require many samples to achieve good performance. Model-based
algorithms, in principle, can provide for much more efficient learning, but
have proven difficult to extend to expressive, high-capacity models such as
deep neural networks. In this work, we demonstrate that medium-sized neural
network models can in fact be combined with model predictive control (MPC) to
achieve excellent sample complexity in a model-based reinforcement learning
algorithm, producing stable and plausible gaits to accomplish various complex
locomotion tasks. We also propose using deep neural network dynamics models to
initialize a model-free learner, in order to com-bine the sample efficiency of
model-based approaches with the high task-specific performance of model-free
methods. We perform this pre-initialization by using rollouts from the trained
model-based controller as supervision to pre-train a policy, and then fine-
tune the policy using a model-free method. We empirically demon-strate that
this resulting hybrid algorithm can drastically accelerate model-free learning
and outperform purely model-free learners on several MuJoCo locomo-tion
benchmark tasks, achieving sample efficiency gains over a purely model-free
learner of 330Ã on swimmer, 26Ã on hopper, 4Ã on half-cheetah, and 3Ã on
ant. Videos can be found at https://sites.google.com/view/mbmf., author =
Nagabandi, Anusha and Kahn, Gregory and Fearing, Ronald S and Levine, Sergey,
booktitle = IEEE International Conference on Robotics and Automation (ICRA),
file = ::, title = Neural Network Dynamics for Model-Based Deep Reinforcement
Learning with Model-Free Fine-Tuning, url =
https://arxiv.org/pdf/1708.02596.pdf, year = 2018 @articleDevin, abstract =
Robotic manipulation in complex open-world scenarios requires both reliable
physical manipulation skills and effective and generalizable perception. In
this paper, we propose a method where general purpose pretrained visual mod-
els serve as an object-centric prior for the perception system of a learned
policy. We devise an object-level attentional mechanism that can be used to
determine rel-evant objects from a few demonstrations, and then immediately
incorporate those objects into a learned policy. A task-independent meta-
attention locates possible objects in the scene, and a task-specific attention
identifies which objects are pre-dictive of the demonstrations. The scope of
the task-specific attention is easily adjusted by showing demonstrations with
distractor objects or with diverse rele-vant objects. Our results indicate
that this approach exhibits good generalization across object instances using
very few samples, and can be used to learn a variety of manipulation tasks
using reinforcement learning., author = Devin, Coline and Abbeel, Pieter and
Darrell, Trevor and Levine, Sergey, file = ::, keywords =
manipulation,reinforcement learning,vision, title = Deep Object-Centric
Representations for Generalizable Robot Learning, url =
https://arxiv.org/pdf/1708.04225.pdf @articleRahmatizadeh, abstract = â
Robots assisting disabled or elderly people in activities of daily living must
perform complex manipulation tasks. These tasks are dependent on the user’s
environment and preferences. Thus, learning from demonstration (LfD) is a
promising choice that would allow the non-expert user to teach the robot
different tasks. Unfortunately, learning general solu-tions from raw
demonstrations requires a significant amount of data. Performing this number
of physical demonstrations is unfeasible for a disabled user. In this paper we
propose an approach where the user demon-strates the manipulation task in a
virtual environment. The collected demonstrations are used to train an LSTM
recurrent neural network that can act as the controller for the robot. We show
that the controller learned from virtual demonstrations can be used to
successfully perform the manipulation tasks on a physical robot., author =
Rahmatizadeh, Rouhollah and Abolghasemi, Pooya and Behal, Aman and Bölöni,
Ladislau, file = ::, title = Learning real manipulation tasks from virtual
demonstrations using LSTM, url = https://arxiv.org/pdf/1603.03833.pdf
@articleWulfmeier, abstract = Training robots for operation in the real world
is a complex, time con-suming and potentially expensive task. Despite
significant success of reinforce-ment learning in games and simulations,
research in real robot applications has not been able to match similar
progress. While sample complexity can be reduced by training policies in
simulation, these can perform sub-optimally on the real platform given
imperfect calibration of model dynamics. We present an approach -supplemental
to fine tuning on the real robot -to further benefit from parallel access to a
simulator during training. The developed approach harnesses auxiliary rewards
to guide the exploration for the real world agent based on the proficiency of
the agent in simulation and vice versa. In this context, we demonstrate em-
pirically that the reciprocal alignment for both agents provides further
benefit as the agent in simulation can adjust to optimize its behaviour for
states commonly visited by the real-world agent., author = Wulfmeier, Markus
and Posner, Ingmar and Abbeel, Pieter, file = ::, keywords = Adversarial
Learning,Robotics,Simulation,Transfer Learning, title = Mutual Alignment
Transfer Learning, url = https://arxiv.org/pdf/1707.07907.pdf @articleSingh,
abstract = We tackle the problem of learning robotic sensorimotor control
policies that can generalize to visually diverse and unseen environments.
Achieving broad generalization typi-cally requires large datasets, which are
difficult to obtain for task-specific interactive processes such as reinforce-
ment learning or learning from demonstration. However, much of the visual
diversity in the world can be captured through passively collected datasets of
images or videos. In our method, which we refer to as GPLAC (Generalized
Policy Learning with Attentional Classifier), we use both interaction data and
weakly labeled image data to augment the generalization capacity of
sensorimotor policies. Our method combines multitask learning on action
selection and an auxiliary binary classification objective, together with a
convolutional neural network architecture that uses an attentional mechanism
to avoid distractors. We show that pairing interaction data from just a single
environment with a diverse dataset of weakly labeled data results in greatly
improved generalization to unseen environments, and show that this
generalization depends on both the auxiliary ob-jective and the attentional
architecture that we propose. We demonstrate our results in both simulation
and on a real robotic manipulator, and demonstrate substantial improve-ment
over standard convolutional architectures and domain adaptation methods.,
author = Singh, Avi and Yang, Larry and Levine, Sergey, file = ::, title =
GPLAC: Generalizing Vision-Based Robotic Skills using Weakly Labeled Images,
url = https://arxiv.org/pdf/1708.02313.pdf @inproceedingsmurali2018cassl,
abstract = Recent self-supervised learning approaches focus on using a few
thou-sand data points to learn policies for high-level, low-dimensional action
spaces. However, scaling this framework for high-dimensional control require
either scal-ing up the data collection efforts or using a clever sampling
strategy for training. We present a novel approach -Curriculum Accelerated
Self-Supervised Learn-ing (CASSL) -to train policies that map visual
information to high-level, higher-dimensional action spaces. CASSL orders the
sampling of training data based on control dimensions: the learning and
sampling are focused on few control pa-rameters before other parameters. The
right curriculum for learning is suggested by variance-based global
sensitivity analysis of the control space. We apply our CASSL framework to
learning how to grasp using an adaptive, underactuated multi-fingered gripper,
a challenging system to control. Our experimental results indicate that CASSL
provides significant improvement and generalization com-pared to baseline
methods such as staged curriculum learning (8% increase) and complete end-to-
end learning with random exploration (14% improvement) tested on a set of
novel objects. Supplementary video: youtube.com/iCQsM7EE4HI., author = Murali,
Adithyavairavan and Pinto, Lerrel and Gandhi, Dhiraj and Gupta, Abhinav,
booktitle = IEEE International Conference on Robotics and Automation (ICRA),
file = ::, title = CASSL: Curriculum Accelerated Self-Supervised Learning, url
= https://arxiv.org/pdf/1708.01354.pdf, year = 2018 @articleVaswani, abstract
= The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks that include an encoder and a decoder. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and
convolutions entirely. Experiments on two machine translation tasks show these
models to be superior in quality while being more parallelizable and requiring
significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014
English-to-German translation task, improving over the existing best results,
including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French
translation task, our model establishes a new single-model state-of-the-art
BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction
of the training costs of the best models from the literature. We show that the
Transformer generalizes well to other tasks by applying it successfully to
English constituency parsing both with large and limited training data.,
author = Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit,
Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Łukasz and Polosukhin,
Illia, file = ::, title = Attention Is All You Need, url =
https://arxiv.org/pdf/1706.03762.pdf @articleOh, abstract = This paper
proposes a novel deep reinforcement learning (RL) architecture, called Value
Prediction Network (VPN), which integrates model-free and model-based RL
methods into a single neural network. In contrast to typical model-based RL
methods, VPN learns a dynamics model whose abstract states are trained to make
option-conditional predictions of future values (discounted sum of re-wards)
rather than of future observations. Our experimental results show that VPN has
several advantages over both model-free and model-based baselines in a
stochastic environment where careful planning is required but building an
accurate observation-prediction model is difficult. Furthermore, VPN
outperforms Deep Q-Network (DQN) on several Atari games even with short-
lookahead planning, demonstrating its potential as a new way of learning a
good state representation., author = Oh, Junhyuk and Singh, Satinder and Lee,
Honglak, file = ::, title = Value Prediction Network, url =
https://arxiv.org/pdf/1707.03497.pdf @articleSchulmanc, abstract = We propose
a new family of policy gradient methods for reinforcement learning, which al-
ternate between sampling data through interaction with the environment, and
optimizing a ” surrogate ” objective function using stochastic gradient
ascent. Whereas standard policy gra-dient methods perform one gradient update
per data sample, we propose a novel objective function that enables multiple
epochs of minibatch updates. The new methods, which we call proximal policy
optimization (PPO), have some of the benefits of trust region policy optimiza-
tion (TRPO), but they are much simpler to implement, more general, and have
better sample complexity (empirically). Our experiments test PPO on a
collection of benchmark tasks, includ-ing simulated robotic locomotion and
Atari game playing, and we show that PPO outperforms other online policy
gradient methods, and overall strikes a favorable balance between sample
complexity, simplicity, and wall-time., author = Schulman, John and Wolski,
Filip and Dhariwal, Prafulla and Radford, Alec and Openai, Oleg Klimov, file =
::, title = Proximal Policy Optimization Algorithms, url =
https://arxiv.org/pdf/1707.06347.pdf
@inproceedingsbellemare2017distributional, abstract = In this paper we argue
for the fundamental impor-tance of the value distribution: the distribution of
the random return received by a reinforcement learning agent. This is in
contrast to the com-mon approach to reinforcement learning which models the
expectation of this return, or value. Although there is an established body of
liter-ature studying the value distribution, thus far it has always been used
for a specific purpose such as implementing risk-aware behaviour. We begin
with theoretical results in both the policy eval-uation and control settings,
exposing a signifi-cant distributional instability in the latter. We then use
the distributional perspective to design a new algorithm which applies
Bellman’s equa-tion to the learning of approximate value distri-butions. We
evaluate our algorithm using the suite of games from the Arcade Learning En-
vironment. We obtain both state-of-the-art re-sults and anecdotal evidence
demonstrating the importance of the value distribution in approxi-mate
reinforcement learning. Finally, we com-bine theoretical and empirical
evidence to high-light the ways in which the value distribution im-pacts
learning in the approximate setting., author = Bellemare, Marc G and Dabney,
Will and Munos, Rémi, booktitle = International Conference on Machine Learning
(ICML), file = ::, title = A Distributional Perspective on Reinforcement
Learning, url = https://arxiv.org/pdf/1707.06887.pdf, year = 2017
@articlepearl1999reasoning, author = Pearl, Judea, file = ::, journal =
International Joint Conference on Artificial Intelligence (IJCAI), title =
Reasoning with Cause and Effect, url =
http://bayes.cs.ucla.edu/IJCAI99/ijcai-99.pdf, year = 1999
@inproceedingsRahmatizadeha, abstract = In this paper, we propose a multi-task
learning from demonstration method that works using raw images as input to
autonomously accomplish a wide variety of tasks in the real world using a low-
cost robotic arm. The controller is a single recurrent neural network that can
generate robot arm trajectories to perform different manipulation tasks. In
order to learn complex skills from relatively few demonstrations, we share
parameters across different tasks. Our network also combines VAE-GAN-based
reconstruction with autoregressive multimodal action prediction for improved
data efficiency. Our results show that weight sharing and reconstruction
substantially improve generalization and robustness, and that training on
multiple tasks simultaneously greatly improves the success rate on all of the
tasks. Our experiments, performed on a real-world low-cost Lynxmotion arm,
illustrate a variety of picking and placing tasks, as well as non-prehensile
manipulation., author = Rahmatizadeh, Rouhollah and Abolghasemi, Pooya and
Bölöni, Ladislau and Levine, Sergey, booktitle = IEEE International Conference
on Robotics and Automation (ICRA), file = ::, keywords = Affordable assistive
robotics,Multi-task learning,Robot manipulation, title = Vision-Based Multi-
Task Manipulation for Inexpensive Robots Using End-To-End Learning from
Demonstration, url = https://arxiv.org/pdf/1707.02920.pdf, year = 2018
@articleDarvish, abstract = The Industry 4.0 paradigm emphasizes the crucial
benefits that collabora-tive robots, i.e., robots able to work alongside and
together with humans, could bring to the whole production process. In this
context, an enabling technol-ogy yet unreached is the design of flexible
robots able to deal at all levels with humans’ intrinsic variability, which is
not only a necessary element for a com-fortable working experience for the
person, but also a precious capability for efficiently dealing with unexpected
events. In this paper, a sensing, represen-tation, planning and control
architecture for flexible human-robot cooperation, referred to as FlexHRC, is
proposed. FlexHRC relies on wearable sensors for human action recognition,
AND/OR graphs for the representation of and rea-soning upon cooperation
models, and a Task Priority framework to decouple action planning from robot
motion planning and control., author = Darvish, Kourosh and Wanderlingh,
Francesco and Bruno, Barbara and Simetti, Enrico and Mastrogiovanni, Fulvio
and Casalino, Giuseppe, file = ::, keywords = AND/OR graph,Human-Robot
Cooperation,Smart Factory,Task Priority control,Wearable Sensing, title =
Flexible Human-Robot Cooperation Models for Assisted Shop-floor Tasks, url =
https://arxiv.org/pdf/1707.02591.pdf @articleLopez-Paz, abstract = One major
obstacle towards artificial intelligence is the poor ability of models to
quickly solve new problems, without forgetting previously acquired knowledge.
To better understand this issue, we study the problem of learning over a
continuum of data, where the model observes, once and one by one, examples
concerning a sequence of tasks. First, we propose a set of metrics to evaluate
models learning over a continuum of data. These metrics characterize models
not only by their test accuracy, but also in terms of their ability to
transfer knowledge across tasks. Second, we propose a model to learn over
continuums of data, called Gradient of Episodic Memory (GEM), which alleviates
forgetting while allowing beneficial transfer of knowledge to previous tasks.
Our experiments on variants of the MNIST and CIFAR-100 datasets demonstrate
the strong performance of GEM when compared to the state-of-the-art., author =
Lopez-Paz, David and Ranzato, Marc Aurelio, file = ::, title = Gradient
Episodic Memory for Continuum Learning, url =
https://arxiv.org/pdf/1706.08840.pdf @articleJustesen, abstract = âThe real-
time strategy game StarCraft has proven to be a challenging environment for
artificial intelligence techniques, and as a result, current state-of-the-art
solutions consist of numerous hand-crafted modules. In this paper, we show how
macromanagement decisions in StarCraft can be learned directly from game
replays using deep learning. Neural networks are trained on 789,571 state-
action pairs extracted from 2,005 replays of highly skilled players, achieving
top-1 and top-3 error rates of 54.6% and 22.9% in predicting the next build
action. By integrating the trained network into UAlbertaBot, an open source
StarCraft bot, the system can significantly outperform the game’s built-in
Terran bot, and play competitively against UAlbertaBot with a fixed rush
strategy. To our knowledge, this is the first time macromanagement tasks are
learned directly from replays in StarCraft. While the best hand-crafted
strategies are still the state-of-the-art, the deep network approach is able
to express a wide range of different strategies and thus improving the
network’s performance further with deep reinforcement learning is an
immediately promising avenue for future research. Ultimately this approach
could lead to strong StarCraft bots that are less reliant on hard-coded
strategies., author = Justesen, Niels and Risi, Sebastian, file = ::, title =
Learning Macromanagement in StarCraft from Replays using Deep Learning, url =
https://njustesen.files.wordpress.com/2017/07/njustesen2017learning.pdf
@articleMerel, abstract = Rapid progress in deep reinforcement learning has
made it increasingly feasible to train controllers for high-dimensional
humanoid bodies. However, methods that use pure reinforcement learning with
simple reward functions tend to pro-duce non-humanlike and overly stereotyped
movement behaviors. In this work, we extend generative adversarial imitation
learning to enable training of generic neural network policies to produce
humanlike movement patterns from limited demonstrations consisting only of
partially observed state features, without access to actions, even when the
demonstrations come from a body with different and unknown physical
parameters. We leverage this approach to build sub-skill policies from motion
capture data and show that they can be reused to solve tasks when controlled
by a higher level controller. [video abstract], author = Merel, Josh and
Tassa, Yuval and Tb, Dhruva and Srinivasan, Sriram and Lemmon, Jay and Wang,
Ziyu and Wayne, Greg and Heess, Nicolas, file = ::, title = Learning human
behaviors from motion capture by adversarial imitation, url =
https://arxiv.org/pdf/1707.02201.pdf @articlecabi2019scaling, title=Scaling
data-driven robotics with reward sketching and batch reinforcement learning,
author=Cabi, Serkan and Colmenarejo, Sergio Gómez and Novikov, Alexander and
Konyushkova, Ksenia and Reed, Scott and Jeong, Rae and Zolna, Konrad and
Aytar, Yusuf and Budden, David and Vecerik, Mel and others, journal=arXiv
preprint arXiv:1909.12200, year=2019 @articleCabi, abstract = This paper
introduces the Intentional Unintentional (IU) agent. This agent endows the
deep deterministic policy gradients (DDPG) agent for contin-uous control with
the ability to solve several tasks simultaneously. Learning to solve many
tasks simultaneously has been a long-standing, core goal of artificial
intelligence, inspired by infant development and motivated by the desire to
build flexible robot manipulators capable of many diverse behaviours. We show
that the IU agent not only learns to solve many tasks simultaneously but it
also learns faster than agents that target a single task at-a-time. In some
cases, where the single task DDPG method completely fails, the IU agent
successfully solves the task. To demonstrate this, we build a playroom
environment using the MuJoCo physics engine, and introduce a grounded formal
language to automatically generate tasks., author = Cabi, Serkan and Gómez
Colmenarejo, Sergio and Hoffman, Matthew W and Denil, Misha and Wang, Ziyu and
De Freitas, Nando, file = ::, keywords = Deep deterministic policy
gradients,control,multi-task,physics, title = The Intentional Unintentional
Agent: Learning to Solve Many Continuous Control Tasks Simultaneously, url =
https://arxiv.org/pdf/1707.03300.pdf @inproceedingsfinn2017maml, abstract = We
propose an algorithm for meta-learning that is model-agnostic, in the sense
that it is com-patible with any model trained with gradient de-scent and
applicable to a variety of different learning problems, including
classification, re-gression, and reinforcement learning. The goal of meta-
learning is to train a model on a vari-ety of learning tasks, such that it can
solve new learning tasks using only a small number of train-ing samples. In
our approach, the parameters of the model are explicitly trained such that a
small number of gradient steps with a small amount of training data from a new
task will produce good generalization performance on that task. In effect, our
method trains the model to be easy to fine-tune. We demonstrate that this
approach leads to state-of-the-art performance on two few-shot image
classification benchmarks, produces good results on few-shot regression, and
acceler-ates fine-tuning for policy gradient reinforcement learning with
neural network policies., author = Finn, Chelsea and Abbeel, Pieter and
Levine, Sergey, booktitle = International Conference on Machine Learning
(ICML), file = ::, title = Model-Agnostic Meta-Learning for Fast Adaptation of
Deep Networks, url = https://arxiv.org/pdf/1703.03400.pdf, year = 2017
@articleLake, abstract = Recent progress in artificial intelligence (AI) has
renewed interest in building systems that learn and think like people. Many
advances have come from using deep neural networks trained end-to-end in tasks
such as object recognition, video games, and board games, achieving perfor-
mance that equals or even beats humans in some respects. Despite their
biological inspiration and performance achievements, these systems differ from
human intelligence in crucial ways. We review progress in cognitive science
suggesting that truly human-like learning and thinking machines will have to
reach beyond current engineering trends in both what they learn, and how they
learn it. Specifically, we argue that these machines should (a) build causal
models of the world that support explanation and understanding, rather than
merely solving pattern recog-nition problems; (b) ground learning in intuitive
theories of physics and psychology, to support and enrich the knowledge that
is learned; and (c) harness compositionality and learning-to-learn to rapidly
acquire and generalize knowledge to new tasks and situations. We suggest
concrete challenges and promising routes towards these goals that can combine
the strengths of recent neural network advances with more structured cognitive
models., author = Lake, Brenden M and Ullman, Tomer D and Tenenbaum, Joshua B
and Gershman, Samuel J, file = ::, title = Building Machines That Learn and
Think Like People, url = https://arxiv.org/pdf/1604.00289.pdf @articleSelsam,
abstract = Noisy data, non-convex objectives, model mis-specification, and
numerical instability can all cause undesired behaviors in machine learning
systems. As a result, detecting actual imple-mentation errors can be extremely
difficult. We demonstrate a methodology in which developers use an interactive
proof assistant to both imple-ment their system and to state a formal theorem
defining what it means for their system to be cor-rect. The process of proving
this theorem inter-actively in the proof assistant exposes all imple-mentation
errors since any error in the program would cause the proof to fail. As a case
study, we implement a new system, Certigrad, for opti-mizing over stochastic
computation graphs, and we generate a formal (i.e. machine-checkable) proof
that the gradients sampled by the system are unbiased estimates of the true
mathematical gradients. We train a variational autoencoder us-ing Certigrad
and find the performance compara-ble to training the same model in
TensorFlow., author = Selsam, Daniel and Liang, Percy and Dill, David L, file
= ::, title = Developing Bug-Free Machine Learning Systems With Formal
Mathematics, url = https://arxiv.org/pdf/1706.08605.pdf
@inproceedingsschaul2016prioritized, abstract = Experience replay lets online
reinforcement learning agents remember and reuse experiences from the past. In
prior work, experience transitions were uniformly sampled from a replay
memory. However, this approach simply replays transitions at the same
frequency that they were originally experienced, regardless of their
significance. In this paper we develop a framework for prioritizing
experience, so as to replay important transitions more frequently, and
therefore learn more efficiently. We use prioritized experience replay in Deep
Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level
performance across many Atari games. DQN with prioritized experience replay
achieves a new state-of-the-art, outperforming DQN with uniform replay on 41
out of 49 games., author = Schaul, Tom and Quan, John and Antonoglou, Ioannis
and Silver, David and Deepmind, Google, booktitle = International Conference
on Learning Representations (ICLR), file = ::, title = Prioritized Experience
Replay, url = https://arxiv.org/pdf/1511.05952.pdf, year = 2016
@articleParisotto, abstract = The ability to act in multiple environments and
transfer previous knowledge to new situations can be considered a critical
aspect of any intelligent agent. To-wards this goal, we define a novel method
of multitask and transfer learning that enables an autonomous agent to learn
how to behave in multiple tasks simultane-ously, and then generalize its
knowledge to new domains. This method, termed ” Actor-Mimic ” , exploits the
use of deep reinforcement learning and model com-pression techniques to train
a single policy network that learns how to act in a set of distinct tasks by
using the guidance of several expert teachers. We then show that the
representations learnt by the deep policy network are capable of general-izing
to new tasks with no prior expert guidance, speeding up learning in novel
environments. Although our method can in general be applied to a wide range of
problems, we use Atari games as a testing environment to demonstrate these
methods., author = Parisotto, Emilio and Ba, Jimmy and Salakhutdinov, Ruslan,
file = ::, title = ACTOR-MIMIC DEEP MULTITASK AND TRANSFER REINFORCEMENT
LEARNING, url = https://arxiv.org/pdf/1511.06342.pdf @articleMorales, abstract
= Fig. 1: Cylindrical robot guiding a box to (purple) goal region.
AbstractâWe address the motion planning problem for a single robot that
manipulates passive objects in an otherwise static environment. We handle
uncertainties in passive object responses while abstracting their dynamics by
creating a roadmap that encodes the possible passive object responses. The
robot extracts reference paths from the passive roadmap to plan approaching
motions with an active roadmap. When the robot acts on the object, it receives
response feedback and reacts accordingly. We apply this approach to pushing, a
basic manipulation primitive. We propose four different pushing strategies,
one of them using reinforcement learning. Our simulations show the
effectiveness of the approach and of each pushing strategy., author = Morales,
Marco and Rodriguez, Samuel and Thomas, Shawna and Amato, Nancy M, file = ::,
title = Sampling-Based Planning and Local Reactive Strategies for Environment
Manipulation, url = http://rss2017ws.is.tuebingen.mpg.de/abstracts/push-
planning-rss17-ws_contact-camera-ready.pdf @articleHosseini, author =
Hosseini, Khaled, file = ::, title = A THOUSAND SPLENDID SUNS, url =
http://english4success.ru/Upload/books/1806.pdf @bookhandmaid, file = ::,
title = The Handmaid’s Tale @articleVidiadharSurajprasadNaipaul, abstract = On
the house in Sikkim Street Mr. Biswas owed, and had been owing for four years,
three thousand dollars. The interest on this, at eight per cent, came to
twenty dollars a month; the ground rent was ten dollars. Two children were at
school. The two older children, on whom Mr. Biswas might have depended, were
both abroad on scholarships. It gave Mr. Biswas some satisfaction that in the
circumstances Shama did not run straight off to her mother to beg for help.
Ten years before that would have been her first thought. Now she tried to
comfort Mr. Biswas, and devised plans on her own. ” Potatoes, ” she said. ” We
can start selling potatoes. The price around here is eight cents a pound. If
we buy at five and sell at sevenâ ” ” Trust the Tulsi bad blood, ” Mr.
Biswas said. ” I know that the pack of you Tulsis are financial geniuses. But
have a good look around and count the number of people selling potatoes.
Better to sell the old car. ” ” No. Not the car. Don’t worry. We’ll manage. ”
” Yes, ” Mr. Biswas said irritably. ” We’ll manage. ” No more was heard of the
potatoes, and Mr. Biswas never threatened again to sell the car. He didn’t now
care to do anything against his wife’s wishes. He had grown to accept her
judgement and to respect her optimism. He trusted her. Since they had moved to
the house Shama had learned a new loyalty, to him and to their children; away
from her mother and sisters, she was able to express this without shame, and
to Mr. Biswas this was a triumph almost as big as the acquiring of his own
house. He thought of the house as his own, though for years it had been
irretrievably mortgaged. And during these months of illness and despair he was
struck again and again by the wonder of being in his own house, the audacity
of it: to walk in through his own front gate, to bar entry to whoever he
wished, to close his doors and windows every night, to hear no noises except
those of his family, to wander freely from room to room and about his yard,
instead of being condemned, as before, to retire the moment he got home to the
crowded room in one or the other of Mrs. Tulsi’s houses, crowded with Shama’s
sisters, their husbands, their children. As a boy he had moved from one house
of strangers to another; and since his marriage he felt he had lived nowhere
but in the houses of the Tulsis, at Hanuman House in Arwacas, in the decaying
wooden house at Shorthills, in the clumsy concrete house in Port of Spain. And
now at the end he found himself in his own house, on his own half-lot of land,
his own portion of the earth. That he should have been responsible for this
seemed to him, in these last months, stupendous. The house could be seen from
two or three streets away and was known all over St. James. It was like a huge
and squat sentry-box: tall, square, two-storeyed, with a pyramidal roof of
corrugated iron. It had been designed and built by a solicitor’s clerk who
built houses in his spare time. The solicitor’s clerk had many contacts. He
bought land which the City Council had announced was not for sale; he
persuaded estate owners to split whole lots into half-lots; he bought lots of
barely reclaimed swamp land near Mucurapo and got permission to build on them.
On whole lots or three-quarter-lots he built one-storey houses, twenty feet by
twenty-six, which could pass unnoticed; on half-lots he built two-storey
houses, twenty feet by thirteen, which were distinctive. All, author =
Vidiadhar Surajprasad Naipaul, By and Riaz Islamabad -Pakistan, Shahid, file =
::, title = A House for Mr. Biswas, url =
http://rgi.edu.in/rgi_pdf/A_House_for_Mr_Biswas_By_Vidiadhar_Surajprasad_Naipaul.pdf
@articleGabrielGarciaMarquez, author = Gabriel Garcia Marquez, By and Riaz
Islamabad -Pakistan, Shahid, file = ::, title = One Hundred Years of Solitude,
url =
http://202.74.245.22:8080/xmlui/bitstream/handle/123456789/164/Marquez.pdf?sequence=1
@articleNarayan, author = Narayan, R K, file = ::, title = A TIGER FOR MALGUDI
PENGUIN BOOKS, url = http://cms.tamindia.com/matrix/images/stories/article-
images/may-2014/A_Tiger_for_Malgudi.pdf @inproceedingszhang2017tensegrity,
abstract = â Tensegrity robots, composed of rigid rods con-nected by elastic
cables, have a number of unique properties that make them appealing for use as
planetary exploration rovers. However, control of tensegrity robots remains a
difficult problem due to their unusual structures and complex dynamics. In
this work, we show how locomotion gaits can be learned automatically using a
novel extension of mirror descent guided policy search (MDGPS) applied to
periodic locomotion move-ments, and we demonstrate the effectiveness of our
approach on tensegrity robot locomotion. We evaluate our method with real-
world and simulated experiments on the SUPERball tensegrity robot, showing
that the learned policies generalize to changes in system parameters,
unreliable sensor measurements, and variation in environmental conditions,
including varied terrains and a range of different gravities. Our experiments
demonstrate that our method not only learns fast, power-efficient feedback
policies for rolling gaits, but that these policies can succeed with only the
limited onboard sensing provided by SUPERball’s accelerometers. We compare the
learned feedback policies to learned open-loop policies and hand-engineered
controllers, and demonstrate that the learned policy enables the first
continuous, reliable locomotion gait for the real SUPERball robot. Our code
and other supplementary materials are available from
http://rll.berkeley.edu/drl_tensegrity, author = Zhang, Marvin and Geng,
Xinyang and Bruce, Jonathan and Caluwaerts, Ken and Vespignani, Massimo and
Sunspiral, Vytas and Abbeel, Pieter and Levine, Sergey, booktitle = IEEE
International Conference on Robotics and Automation (ICRA), file = ::, title =
Deep Reinforcement Learning for Tensegrity Robot Locomotion, url =
https://arxiv.org/pdf/1609.09049.pdf, year = 2017
@inproceedingsbacon2017optioncritic, abstract = Temporal abstraction is key to
scaling up learning and plan-ning in reinforcement learning. While planning
with tempo-rally extended actions is well understood, creating such ab-
stractions autonomously from data has remained challenging. We tackle this
problem in the framework of options [Sutton, Precup & Singh, 1999; Precup,
2000]. We derive policy gra-dient theorems for options and propose a new
option-critic architecture capable of learning both the internal policies and
the termination conditions of options, in tandem with the pol-icy over
options, and without the need to provide any addi-tional rewards or subgoals.
Experimental results in both dis-crete and continuous environments showcase
the flexibility and efficiency of the framework., author = Bacon, Pierre-Luc
and Harb, Jean and Precup, Doina, booktitle = AAAI Conference on Artificial
Intelligence, file = ::, title = The Option-Critic Architecture, url =
https://arxiv.org/pdf/1609.05140.pdf, year = 2017 @articleSzita, abstract =
The exploration-exploitation dilemma has been an intriguing and unsolved
problem within the framework of reinforcement learn-ing. ” Optimism in the
face of uncertainty ” and model building play central roles in ad-vanced
exploration methods. Here, we inte-grate several concepts and obtain a fast
and simple algorithm. We show that the proposed algorithm finds a near-optimal
policy in poly-nomial time, and give experimental evidence that it is robust
and efficient compared to its ascendants., author = Szita, István, file = ::,
title = The Many Faces of Optimism: a Unifying Approach, url =
http://icml2008.cs.helsinki.fi/papers/490.pdf @articleSingha, abstract =
Reinforcement learning has achieved broad and successful ap-plication in
cognitive science in part because of its general for-mulation of the adaptive
control problem as the maximization of a scalar reward function. The
computational reinforcement learning framework is motivated by correspondences
to ani-mal reward processes, but it leaves the source and nature of the
rewards unspecified. This paper advances a general computa-tional framework
for reward that places it in an evolutionary context, formulating a notion of
an optimal reward function given a fitness function and some distribution of
environments. Novel results from computational experiments show how tra-
ditional notions of extrinsically and intrinsically motivated be-haviors may
emerge from such optimal reward functions. In the experiments these rewards
are discovered through auto-mated search rather than crafted by hand. The
precise form of the optimal reward functions need not bear a direct
relationship to the fitness function, but may nonetheless confer significant
advantages over rewards based only on fitness., author = Singh, Satinder and
Lewis, Richard L and Barto, Andrew G, file = ::, title = Where Do Rewards Come
From?, url = https://web.eecs.umich.edu/ baveja/Papers/singh-lewis-
barto-2009-cogsci.pdf @articleKendall, abstract = Deep learning has shown to
be effective for robust and real-time monocular image relocalisation. In
particular, PoseNet [22] is a deep convolutional neural network which learns
to regress the 6-DOF camera pose from a single im-age. It learns to localize
using high level features and is robust to difficult lighting, motion blur and
unknown cam-era intrinsics, where point based SIFT registration fails.
However, it was trained using a naive loss function, with hyper-parameters
which require expensive tuning. In this paper, we give the problem a more
fundamental theoreti-cal treatment. We explore a number of novel loss
functions for learning camera pose which are based on geometry and scene
reprojection error. Additionally we show how to au-tomatically learn an
optimal weighting to simultaneously regress position and orientation. By
leveraging geometry, we demonstrate that our technique significantly improves
PoseNet’s performance across datasets ranging from indoor rooms to a small
city., author = Kendall, Alex and Cipolla, Roberto, file = ::, title =
Geometric loss functions for camera pose regression with deep learning, url =
https://arxiv.org/pdf/1704.00390.pdf @articleZenke, abstract = While deep
learning has led to remarkable ad-vances across diverse applications, it
struggles in domains where the data distribution changes over the course of
learning. In stark contrast, biological neural networks continually adapt to
changing domains, possibly by leveraging com-plex molecular machinery to solve
many tasks simultaneously. In this study, we introduce in-telligent synapses
that bring some of this bio-logical complexity into artificial neural
networks. Each synapse accumulates task relevant informa-tion over time, and
exploits this information to rapidly store new memories without forgetting old
ones. We evaluate our approach on continual learning of classification tasks,
and show that it dramatically reduces forgetting while maintain-ing
computational efficiency., author = Zenke, Friedemann and Poole, Ben and
Ganguli, Surya, file = ::, title = Continual Learning Through Synaptic
Intelligence, url = https://arxiv.org/pdf/1703.04200.pdf @articleAmodei,
abstract = Rapid progress in machine learning and artificial intelligence (AI)
has brought increasing atten-tion to the potential impacts of AI technologies
on society. In this paper we discuss one such potential impact: the problem of
accidents in machine learning systems, defined as unintended and harmful
behavior that may emerge from poor design of real-world AI systems. We present
a list of five practical research problems related to accident risk,
categorized according to whether the problem originates from having the wrong
objective function (” avoiding side effects ” and ” avoiding reward hacking
”), an objective function that is too expensive to evaluate frequently (”
scalable supervision ”), or undesirable behavior during the learning process
(” safe exploration ” and ” distributional shift ”). We review previous work
in these areas as well as suggesting re-search directions with a focus on
relevance to cutting-edge AI systems. Finally, we consider the high-level
question of how to think most productively about the safety of forward-looking
applications of AI., author = Amodei, Dario and Olah, Chris and Steinhardt,
Jacob and Christiano, Paul and Openai, John Schulman and Mané, Dan and Brain,
Google, file = ::, title = Concrete Problems in AI Safety, url =
https://arxiv.org/pdf/1606.06565.pdf @articleKirkpatrick, abstract = The
ability to learn tasks in a sequential fashion is crucial to the development
of artificial intelligence. Neural networks are not, in general, capable of
this and it has been widely thought that catastrophic forgetting is an
inevitable feature of connectionist models. We show that it is possible to
overcome this limitation and train networks that can maintain expertise on
tasks which they have not experienced for a long time. Our approach remembers
old tasks by selectively slowing down learning on the weights important for
those tasks. We demonstrate our approach is scalable and effective by solving
a set of classification tasks based on the MNIST hand written digit dataset
and by learning several Atari 2600 games sequentially., author = Kirkpatrick,
James and Pascanu, Razvan and Rabinowitz, Neil and Veness, Joel and
Desjardins, Guillaume and Rusu, Andrei A and Milan, Kieran and Quan, John and
Ramalho, Tiago and Grabska-Barwinska, Agnieszka and Hassabis, Demis and
Clopath, Claudia and Kumaran, Dharshan and Hadsell, Raia, file = ::, title =
Overcoming catastrophic forgetting in neural networks, url =
https://arxiv.org/pdf/1612.00796.pdf @articlekroemer2010grasping, abstract =
Grasping an ob jet is a task that inherently needs to be treated in a hybrid
fashion. The system must deide both where and how to grasp the ob jet. While
seleting where to grasp requires learning about the ob jet as a whole, the
exeution only needs to reatively adapt to the ontext lose to the grasp’s
loation. We propose a hierarhial ontroller that reÆets the struture of these
two sub-problems, and attempts to learn solutions that work for both. A hybrid
arhiteture is employed by the ontroller to make use of various mahine learning
methods that an ope with the large amount of unertainty inherent to the task.
The ontroller’s upper level selets where to grasp the ob jet using a
reinforement learner, while the lower level omprises an imitation learner and
a vision-based reative ontroller to determine appropriate grasping motions.
The resulting system is able to quikly learn good grasps of a novel ob jet in
an unstrutured environment, by exeuting smooth reahing motions and preshaping
the hand depending on the ob jet’s geometry. The system was evaluated both in
simulation and on a real robot., author = Kroemer, Oliver and Detry, Renaud
and Piater, Justus and Peters, Jan, file = ::, journal = Robotics and
Autonomous systems, mendeley-groups = Old Research, number = 9, pages =
1105–1116, title = Combining Active Learning and Reactive Control for Robot
Grasping, url =
https://orbi.uliege.be//bitstream/2268/60643/1/Kroemer-2010-RAS-author-
postprint.pdf, volume = 58, year = 2010 @articleGoodfellow, abstract =
Catastrophic forgetting is a problem faced by many machine learning models and
al-gorithms. When trained on one task, then trained on a second task, many
machine learning models ” forget ” how to perform the first task. This is
widely believed to be a serious problem for neural networks. Here, we
investigate the extent to which the catas-trophic forgetting problem occurs
for mod-ern neural networks, comparing both estab-lished and recent gradient-
based training al-gorithms and activation functions. We also examine the
effect of the relationship between the first task and the second task on
catas-trophic forgetting. We find that it is always best to train using the
dropout algorithmâ the dropout algorithm is consistently best at adapting to
the new task, remembering the old task, and has the best tradeoff curve be-
tween these two extremes. We find that dif-ferent tasks and relationships
between tasks result in very different rankings of activation function
performance. This suggests that the choice of activation function should
always be cross-validated., author = Goodfellow, Ian J and Mirza, Mehdi and
Xiao, Da and Courville, Aaron and Bengio, Yoshua, file = ::, title = An
Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural
Networks, url = https://arxiv.org/pdf/1312.6211.pdf
@inproceedingsruvolo2013ella, abstract = The problem of learning multiple
consecu-tive tasks, known as lifelong learning, is of great importance to the
creation of intelli-gent, general-purpose, and flexible machines. In this
paper, we develop a method for on-line multi-task learning in the lifelong
learn-ing setting. The proposed Efficient Life-long Learning Algorithm (ELLA)
maintains a sparsely shared basis for all task models, transfers knowledge
from the basis to learn each new task, and refines the basis over time to
maximize performance across all tasks. We show that ELLA has strong
connections to both online dictionary learning for sparse coding and state-of-
the-art batch multi-task learning methods, and provide robust the-oretical
performance guarantees. We show empirically that ELLA yields nearly identi-cal
performance to batch multi-task learning while learning tasks sequentially in
three or-ders of magnitude (over 1,000x) less time., author = Ruvolo, Paul and
Eaton, Eric, booktitle = International Conference on Machine Learning (ICML),
file = ::, title = ELLA: An Efficient Lifelong Learning Algorithm, url =
http://proceedings.mlr.press/v28/ruvolo13.pdf, year = 2013
@inproceedingswilliams2017mpc, abstract = — We introduce an information
theoretic model pre-dictive control (MPC) algorithm capable of handling
complex cost criteria and general nonlinear dynamics. The generality of the
approach makes it possible to use multi-layer neural networks as dynamics
models, which we incorporate into our MPC algorithm in order to solve model-
based reinforcement learning tasks. We test the algorithm in simulation on a
cart-pole swing up and quadrotor navigation task, as well as on actual
hardware in an aggressive driving task. Empirical results demonstrate that the
algorithm is capable of achieving a high level of performance and does so only
utilizing data collected from the system., author = Williams, Grady and
Wagener, Nolan and Goldfain, Brian and Drews, Paul and Rehg, James M and
Boots, Byron and Theodorou, Evangelos A, booktitle = International Conference
on Robotics and Automation (ICRA), file = ::, title = Information Theoretic
MPC for Model-Based Reinforcement Learning, url = http://www.cc.gatech.edu/
bboots3/files/InformationTheoreticMPC.pdf, year = 2017 @articleMontgomery,
abstract = Guided policy search algorithms can be used to optimize complex
nonlinear poli-cies, such as deep neural networks, without directly computing
policy gradients in the high-dimensional parameter space. Instead, these
methods use supervised learning to train the policy to mimic a ” teacher ”
algorithm, such as a trajectory optimizer or a trajectory-centric
reinforcement learning method. Guided policy search methods provide asymptotic
local convergence guarantees by construction, but it is not clear how much the
policy improves within a small, finite number of iterations. We show that
guided policy search algorithms can be interpreted as an approximate variant
of mirror descent, where the projection onto the constraint manifold is not
exact. We derive a new guided policy search algorithm that is sim-pler and
provides appealing improvement and convergence guarantees in simplified convex
and linear settings, and show that in the more general nonlinear setting, the
error in the projection step can be bounded. We provide empirical results on
several simulated robotic navigation and manipulation tasks that show that our
method is stable and achieves similar or better performance when compared to
prior guided policy search methods, with a simpler formulation and fewer
hyperparameters., author = Montgomery, William and Levine, Sergey, file = ::,
title = Guided Policy Search as Approximate Mirror Descent, url =
https://arxiv.org/pdf/1607.04614.pdf @inproceedingschebotar2017pilqr, abstract
= Reinforcement learning (RL) algorithms for real-world robotic applications
need a data-efficient learning process and the ability to handle com-plex,
unknown dynamical systems. These re-quirements are handled well by model-based
and model-free RL approaches, respectively. In this work, we aim to combine
the advantages of these two types of methods in a principled manner. By
focusing on time-varying linear-Gaussian poli-cies, we enable a model-based
algorithm based on the linear quadratic regulator (LQR) that can be integrated
into the model-free framework of path integral policy improvement (PI 2). We
can further combine our method with guided pol-icy search (GPS) to train
arbitrary parameterized policies such as deep neural networks. Our sim-ulation
and real-world experiments demonstrate that this method can solve challenging
manipula-tion tasks with comparable or better performance than model-free
methods while maintaining the sample efficiency of model-based methods.,
author = Chebotar, Yevgen and Hausman, Karol and Zhang, Marvin and Sukhatme,
Gaurav and Schaal, Stefan and Levine, Sergey, booktitle = International
Conference on Machine Learning (ICML), file = ::, title = Combining Model-
Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning,
url = https://arxiv.org/pdf/1703.03078.pdf, year = 2017 @articleKlambauer,
abstract = Deep Learning has revolutionized vision via convolutional neural
networks (CNNs) and natural language processing via recurrent neural networks
(RNNs). However, success stories of Deep Learning with standard feed-forward
neural networks (FNNs) are rare. FNNs that perform well are typically shallow
and, therefore cannot exploit many levels of abstract representations. We
introduce self-normalizing neural networks (SNNs) to enable high-level
abstract representations. While batch normalization requires explicit
normalization, neuron activations of SNNs automatically converge towards zero
mean and unit variance. The activation function of SNNs are ” scaled
exponential linear units ” (SELUs), which induce self-normalizing properties.
Using the Banach fixed-point theorem, we prove that activations close to zero
mean and unit variance that are propagated through many network layers will
converge towards zero mean and unit variance â even under the presence of
noise and perturbations. This convergence property of SNNs allows to (1) train
deep networks with many layers, (2) employ strong regularization schemes, and
(3) to make learning highly robust. Furthermore, for activations not close to
unit variance, we prove an upper and lower bound on the variance, thus,
vanishing and exploding gradients are impossible. We compared SNNs on (a) 121
tasks from the UCI machine learning repository, on (b) drug discovery
benchmarks, and on (c) astronomy tasks with standard FNNs, and other machine
learning methods such as random forests and support vector machines. For FNNs
we considered (i) ReLU networks without normalization, (ii) batch
normalization, (iii) layer normalization, (iv) weight normalization, (v)
highway networks, and (vi) residual networks. SNNs significantly outperformed
all competing FNN methods at 121 UCI tasks, outperformed all competing methods
at the Tox21 dataset, and set a new record at an astronomy data set. The
winning SNN architectures are often very deep. Implementations are available
at: github.com/bioinf-jku/SNNs., author = Klambauer, Günter and Unterthiner,
Thomas and Mayr, Andreas and Hochreiter, Sepp, file = ::, title = Self-
Normalizing Neural Networks, url = https://arxiv.org/pdf/1706.02515.pdf
@articleWatters, abstract = From just a glance, humans can make rich
predictions about the future state of a wide range of physical systems. On the
other hand, modern approaches from engineering, robotics, and graphics are
often restricted to narrow domains and require direct measurements of the
underlying states. We introduce the Visual Interaction Network, a general-
purpose model for learning the dynamics of a physical system from raw visual