# Towards Offloadable and Migratable Microservices on Disaggregated
Architectures: Vision, Challenges, and Research Roadmap

Xiaoyi Lu and Arjun Kashyap
_University of California, Merced_
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Microservice and serverless computing systems open up massive versatility and
opportunity to distributed and datacenter-scale computing. In the meantime,
the deployments of modern datacenter resources are moving to disaggregated
architectures. With the flourishing growths from both sides, we believe this
is high time to write this vision paper to propose a potential research agenda
to achieve efficient deployments, management, and executions of next-
generation microservices on top of the emerging disaggregated datacenter
architectures. In particular, we envision a critical systems research
direction of designing and developing offloadable and migratable microservices
on disaggregated architectures. With this vision, we have surveyed the recent
related work to demonstrate the importance and necessity of researching it. We
also outline the fundamental challenges that distributed systems and
datacenter-scale computing research may encounter. We further propose a
research roadmap to achieve our envisioned objectives in a promising way.
Within the roadmap, we identify potential techniques and methods that can be
leveraged.

##  1 Introduction

Microservices and serverless computing are becoming the new paradigm in cloud
computing. The microservice architecture of decoupling monolithic applications
to loosely-coupled microservices is gaining traction in large datacenters due
to the ability to meet strict performance and availability constraints. Big
cloud providers have already started adopting the microservice application
model [5, 8, 9]. Similarly, many public cloud platforms are offering
serverless computing services like AWS Lambda [2], Azure Functions [3], and
Google Cloud Functions [6] as the platform automatically manages the
infrastructure and server to run these serverless functions.

With the increasing popularity of microservices, current efforts have been put
to study the characteristics, requirements, and implications they have on the
cloud system stack [17, 30, 43, 46, 51]. Due to the strict service level
objectives set (Quality-of-Service requirements) by customers for throughput
and tail latency, microservices put pressure on cloud architecture ranging
from hardware to operating system and all the way up to application design.
Lastly, datacenters have a huge energy footprint [33] and these days the cloud
providers equally focus on both performance and energy-efficiency of the
applications to reduce the carbon footprint.

On the other hand, Disaggregated Architecture for modern datacenters becomes a
hot research topic recently. Disaggregated architecture splits the existing
monolithic datacenter tier into at least two independent resource pools (i.e.,
compute pool and storage pool) that communicate over fast interconnects, such
as high-speed Ethernet or InfiniBand. By sharing the storage pool across
diverse datacenter services, disaggregated architecture consolidates
computational and storage resources independently. Thus, it allows independent
optimization and resource scaling, and reduces total cost. The cost and
efficiency benefits of disaggregated architecture have attracted interest from
both academia and industry [26, 16, 31, 44, 41]. Modern public cloud providers
such as Amazon and Microsoft Azure Cloud [13] have already adopted high-level
compute and storage disaggregation.

With these trends, more and more microservices or serverless computing-based
applications will be deployed and run on top of disaggregated architectures of
modern and next-generation datacenters [38]. In this kind of deployments, we
envision that next-generation microservices will be more often offloaded and
executed on remote computational devices or disaggregated resources. In the
meantime, these microservices need to be migratable across different layers of
heterogeneous hardware platforms, due to the requirements of manageability,
performance, load balancing, and so on. In other words, we believe that
offloadable and migratable microservices will become a key computing paradigm
on disaggregated datacenter architectures in the future.

In this paper, we first present a brief survey on related work (Section 2) to
give a high-level overview of existing studies in related research directions.
Then, we outline a vision (Section 3) for demonstrating how and why
microservices should be offloaded and migratable on disaggregated datacenter
architectures. The associated challenges that distributed systems research may
encounter are discussed in Section 4. To address these fundamental challenges,
we propose a roadmap (Section 5), mapping key research challenges to bodies of
work that have the potential to realize high-performance, scalable, energy-
efficient, and QoS-aware offloadable and migratable microservice systems on
disaggregated architectures. The key contributions of this paper are:

  * •

Envisioned a critical systems research direction of designing and developing
next-generation offloadable and migratable microservices on emerging
disaggregated datacenter architectures.

  * •

Surveyed the recent related studies in the literature and the community to
demonstrate the importance and necessity of this research direction.

  * •

Outlined a vision and the associated fundamental challenges that distributed
systems and datacenter-scale computing research may encounter when building
next-generation offloadable and migratable microservices.

  * •

Proposed a research roadmap to achieve our envisioned objectives in a
promising way. Within the roadmap, we identify potential techniques and
methods that can be leveraged.

##  2 A Brief Survey on Related Work

This section presents a brief survey on recent studies, which are closely
related to our research agenda.

###  2.1 Microservice Benchmarking

Research has been done on evaluating and benchmarking microservices [17, 30,
43, 46, 51, 19] but none of them discuss the pros and cons when making
microservices offloadable and migratable. DeathStarBench [17] contains
microservices that are representative of large end-to-end services and studies
their implications across the cloud system stack. It also shows that running
microservices on low power cores leads to slight performance degradation but
saves power. Thus, these cores could be utilized for microservices that are
off the critical path. µSuite [43] investigates the impact of network and OS
overheads on microservice median and tail latency. Ueda et al. [46] present
the impact of language runtime and hardware architecture on microservice
performance. Zhou et al. [51] identify the gaps between existing benchmark
systems and industrial microservice systems and develop a new microservice
benchmarking system. ppbench [30] proposes a benchmark to study the
performance impact of programming languages and networks on microservices.

###  2.2 Disaggregated Datacenter Architecture

Disaggregated datacenter architecture has many advantages like flexibility and
independent scaling of resources (compute, network, and/or storage), fine-
grained resource provisioning, and higher utilization. We believe this
architecture will continue to evolve and include new computational devices
which microservices should benefit from. A lot of work has been done in the
community to allow applications to reap the benefits of resource
disaggregation [11, 42, 41, 35, 32, 31]. LegoOS [41] is a new OS model for
hardware resource disaggregation that splits traditional operating system
functionalities into loosely-coupled units. soNUMA [35] enables low latency
and distributed in-memory processing by eliminating data movement across
network stack and PCIe interface. Legtchenko et al. [31] discuss the concept
of disaggregated storage at rack-level and propose a novel storage fabric for
different disaggregation types for cloud storage.

###  2.3 Near-Data Processing

With the recent trends in disaggregated datacenter architectures and pushing
towards improving energy-efficiency, many applications are being re-designed
to move compute near to memory/storage. Near-Data Processing (NDP) has been
explored in the areas of databases [12, 14, 25, 36], computer vision [24],
machine-learning inference [49], big-data systems [20], and key-value stores
[15, 29, 47] with the vision to reduce data movement. We believe microservices
should also exploit this paradigm. Hu et al. [24] present a storage system
that adjusts the dataset resolution within the storage device by utilizing
idle SSD cores that are otherwise used by flash translation layer and garbage
collection inside an SSD. E3 [33] is a microservice execution platform that
offloads microservices to low-power SmartNIC processors to improve energy-
efficiency without much loss in latency. E3 also migrates microservices to
host to eliminate SmartNIC overloading via a communication-aware service
placement algorithm. Though some of our motivations are similar to prior work,
e.g., reducing energy consumption on heterogeneous compute substrates, we not
only want microservices to be migratable to any available compute abstraction
in the cluster but also to benefit from NDP by leveraging computational
devices in future datacenter architectures without losing on the productivity
of developers.

###  2.4 Resource Virtualization & Management

Microservices are generally built on top of resource virtualization techniques
like containers and virtual machines (VMs). Apache OpenWhisk [1] is an open-
source, distributed serverless platform that executes functions in response to
events at any scale. OpenWhisk manages the infrastructure, servers, and
scaling using Docker containers. It supports many container frameworks and
programming languages in which one can write functional logic and run them in
response to events or from HTTP requests.

I/O virtualization introduces a major bottleneck for high-performance
networks. Remote Direct Memory Access (RDMA) virtualization is one way to
achieve migration of microservices in the new datacenter architecture across
diverse compute units. Current work in this space, namely, SR-IOV [37, 27],
LITE [45], vSocket [48], HyV [39], and FreeFlow [28] have heavy connection
setup costs and may not be suitable for container-based virtualized
environments. MasQ [23] is a software-defined RDMA virtualization technique to
allow VMs in public clouds to realize virtual RoCE devices. MigrOS [40] allows
transparent live migration for RDMA-enabled containers.

##  3 Vision

As shown in Figure 1, we envision the next-generation disaggregated datacenter
architectures to comprise of persistent memory [50, 22] (PMEM), computational
PMEM (CPM) [7], NVMe-SSDs, computational SSDs (CSDs) [4], RDMA-enabled
networks, and remote memory [10, 21]. Due to the increasing opportunities of
co-designing systems between hardware and software, we want the microservices
and serverless functions to be able to take advantage of these new emerging
hardware architectures in datacenters. Future microservice systems should
allow microservices and serverless applications to leverage this new
datacenter architecture by co-designing applications and their underlying
systems and runtime. This will improve their performance, scalability, energy-
efficiency, QoS support while maintaining similar productivity of the
microservices developer.

Figure 1: Next-generation disaggregated datacenter architectures. M1, M2, and
M3 are the microservices that were initially running on compute nodes and get
offloaded to computational devices to reduce data movement and achieve high
performance, energy-efficiency, and other benefits.

There are a lot of low-level challenges in co-designing microservice
applications and frameworks with the emerging hardware. First, microservices
and serverless functions could be made up of different programming models and
languages adding to system heterogeneity and supporting them in diverse
hardware platforms becomes cumbersome. Second, as the computational storage
devices, like CSD and CPM, have limited compute and memory capacity (like ARM-
based SoCs) compared to x86 servers, utilizing their resources efficiently
without degrading their main functions is hard. Data-intensive microservices
that have less on-device memory consumption (like data scanning and filtering)
may be more appropriate to be offloaded to CSDs and CPMs. Third, with the
introduction of new devices, application requirements of performance,
scalability, QoS, productivity, and energy-efficiency should be met. A
significant amount of energy and execution time consumed during an
application’s lifetime goes into data movement between the compute and memory
[18, 47, 14]. Thus, the key insight here is to reduce this data movement by
pushing/offloading a certain amount of computing in microservices or whole
microservices near these computational devices.

Once microservices are offloaded on disaggregated resources, we will
immediately meet another requirement that is supporting migratable
microservices. This is because depending upon load distribution, data
distribution, and resource availability, microservices may need to be
rebalanced and migrated to other remote resources. As shown in Figure 1,
microservices (e.g., M1, M2, and M3) were initially running on local compute
nodes. Later, they may be offloaded to some local or remote computational
devices (like CPM or CSD) due to data locality. At some point, M1 or M3 may
get migrated to CPM devices from ARM-based CSD platforms to get better
performance, energy efficiency, and load balancing.

We envision that many application scenarios can easily get benefits from
offloadable and migratable microservices on disaggregated architectures.
Taking machine learning pipelines as an example, modern machine learning
pipelines including training and inference stages need not only a huge amount
of computing resources such as GPGPUs, TPUs, or ASICs but also data storage
and movement over networks and storage devices. A well-designed distributed
system such as Ray [34] can significantly improve the performance,
manageability, and flexibility of machine learning pipelines. Machine learning
pipelines can be encapsulated to different microservices to get better
manageability and flexibility. In this paper, we envision that if next-
generation microservice management systems can smartly offload and migrate
microservices, then we can efficiently distribute training microservices, pre-
processing microservices, inferencing microservices, and post-processing
microservices in machine learning pipelines to the ‘best place’ to be
executed. In this way, machine learning pipelines with our proposed
microservices can significantly reduce data movements, execution times, energy
consumption, and so on.

##  4 Challenges

As indicated earlier, the availability of computational devices near storage
and memory, like CSDs and CPM, and next-generation datacenter platforms
provide novel opportunities to co-design microservices/serverless applications
and their underlying runtime to significantly improve energy-efficiency,
performance, scalability, QoS, and so on. However, much of the work to utilize
computational devices is still application-specific [47, 14, 7] and cannot be
used by general-purpose microservices or serverless functions.

Data-intensive applications spend a significant amount of time and energy to
move data towards compute. But with the availability of compute units near the
memory and storage devices, certain tasks can be offloaded to these devices to
save energy consumption and improve the performance of the application. Thus,
it becomes critical to co-design microservice applications to take advantage
of the next-generation datacenter hardware to reduce data movement and lessen
their carbon footprint. Based on these trends, we envision that hardware
evolution and cross-layer software co-design will continue to be the two most
important driving forces for the architectural evolution and performance-
boosting of future serverless computing systems.

Based on these observations, we identify several fundamental research
challenges to achieve the goals of offloadable and migratable
microservices/serverless functions, which aim to fully utilize the provided
resources on modern and next-generation datacenter infrastructures.

  1. 1.

How can we design effective and efficient runtime systems for microservices
and serverless functions that can take advantage of the novel features of
next-generation datacenter hardware?

  2. 2.

How can we co-design with applications and microservice platforms to achieve
near-native performance while reducing energy consumption?

  3. 3.

How to offload and migrate the entire microservice or a portion of it to
computational devices in an energy-efficient and QoS-aware way without
burdening any of resource-constrained devices?

  4. 4.

How to offload such tasks with minimal code changes to the applications to
keep the developer productivity high?

  5. 5.

What kind of benefits can be achieved through these new designs? How to
benchmark these designs with microservice platforms and applications?

##  5 Research Roadmap

This section presents our proposed research roadmap as shown in Figure 2.

(a)  (b)

Figure 2: Proposed Research Roadmap

###  5.1 Standardized Programmable Hardware Interface

First of all, it is a challenging task to offload an entire or a portion of a
microservice to heterogeneous compute units in the datacenter. This is because
we need a standardized, generic, and programmable interface from the
underlying hardware layer. Creating such an interface is challenging in a
disaggregated datacenter architecture as the heterogeneous computational
devices could be interconnected via the memory bus, intra-node, or inter-node
interconnects (like PCIe, CXL, Gen-Z, NVLink, RoCE, InfiniBand, etc.).
Interacting with these devices typically needs to change systems or
applications a lot currently. Hence, future research in this space should
target offloading any functionality across the datacenter in a way that has
minimal impact on the productivity of the developer and supports both built-in
and resource-conservative user-defined functions. Even though this is a
difficult research task, we do see success stories in the community. For
instance, RDMA with verbs and NVMe protocols are becoming standardized
programmable interfaces for high-speed networks and storage devices. We
envision that more standardized programmable interfaces will become available
in the future.

###  5.2 Portable Compute Abstractions

When migrating/offloading microservices on various compute devices in the
datacenter, the ideal technique would be not to rewrite/recompile code for
each device. For instance, it may be a difficult problem to avoid the cross-
compilation issue when running microservices on different compute devices. One
possible solution is we could consider generating a number of binaries and
using the correct one on each compute device.

In addition, producing new compute abstractions would help us in reaching this
goal. Such abstractions should minimize the compute footprint in order to
reduce the compute movement cost across the datacenter and allow microservices
to transfer/share the output of offloaded compute. Thus, the community should
propose new abstractions which not only increase productivity but also
maintain portability and performance. So far, the most common ways to deploy
microservices are based on virtual machines or containers. To solve all the
challenges we listed earlier, it may be an interesting and open challenge to
rethink what could be the best way to design portable compute abstractions for
microservices.

###  5.3 Offload-aware Systems, Runtimes, and APIs

Microservices could run numerous kinds of workloads, which can be both
stateful and stateless workloads. We foresee different approaches should be
adopted to handle stateful and stateless microservices. In order to fully
support various workloads and execute offloaded functions on different kinds
of computational devices, we need offload-aware systems, runtimes, and APIs.
Developing such kind of systems to fully leverage diverse compute units in an
application-agnostic manner and at the same time add minimal overhead is an
interesting research direction. For example, an offload-ware communication and
I/O runtime should be able to automatically and transparently select the best
protocols for microservice applications based on the capabilities of the
devices where the microservices are offloaded to.

###  5.4 Live Migration Capability

Microservice migration, especially for stateful microservices, is important to
avoid overloading and achieve good utilization across computational devices in
a datacenter. Migration needs not to be just from one host/server to another.
It could be, for example, from a local computational storage device to a
remote computational persistent memory device or the other way around. We need
to develop new mechanisms that enable fast live migration of microservices
using different network protocols and interconnects while preserving
performance. Another important aspect of enabling live migration would be to
figure out which heterogeneous compute unit has the resources to support the
migrating microservice without violating the QoS support.

###  5.5 Task-Aware Near Data Offload

For maximum utilization of available compute resources and to reduce data
movement and energy consumption, task and capability aware offloading of
microservices is necessary. A good offload strategy should decide where to
offload and also take into consideration offloading costs vs. rewards. Future
research in task-aware offloading should explore designing new metrics and
models which could characterize and predict when and on which computational
device it would be useful to offload microservices in disaggregated
datacenters. The potential modeling and decision-making approaches can be
built with some artificial intelligence techniques, such as reinforcement
learning.

###  5.6 Dynamic Resource Partitioning

Due to the heterogeneous nature of compute, memory, and workloads on a
computational device in a datacenter, dynamic resource partitioning for
offloaded functions is a challenge. One must ensure that running microservices
or functions on CSDs/CPMs should not impact the performance of their main task
– I/O processing. To achieve this, hardware vendors may need to provide more
resources or mechanisms on hardware devices to help systems designs. For
example, dynamic I/O queues or resource pools can be provided from the
hardware layer to achieve better designs.

###  5.7 Isolation and Load Balancing

As these computational devices will run multiple microservices/offloaded
functions, there should be strong security and isolation guarantees among them
as present on a host server. Some possible technology examples include
hardware-based data encryption, namespaces, protection domains, etc.

Also, the load balancer which would analyze the load of all or nearby devices
should make offloading decisions based on data locality and device load. To
achieve scalable load balancing, both centralized and decentralized load
balancing algorithms should be designed.

###  5.8 Energy and QoS Aware Systems

All microservices have to meet their Service Level Objectives (SLOs), so
offloading of microservices, especially compute-intensive in nature, should
occur if running them on computational devices does not violate their QoS
guarantees. Another important factor to consider while executing microservices
is to respect the pre-defined energy/power budget. Hence, the challenge would
be to balance the performance-energy trade-off based on varying application
and user needs. These goals can be achieved through accurate modeling and
enforcement in microservice systems. Artificial intelligence techniques may
again play a very important role in this research direction.

###  5.9 Co-design Hardware & Software

Multiple layers of software frameworks, programming model semantics, and newer
architectures have rendered the design process to be complicated and isolated.
Thus, a holistic cross-layer approach is critical to tackling the challenges
at these layers. The community needs to develop runtimes or systems that could
provide efficient virtualization mechanisms, live migration capability, and
support all the other research challenges outlined above. Then microservices
and serverless functions could be co-designed to coordinate and interact with
the underlying runtimes and reap all the benefits of next-generation
datacenter infrastructures.

We envision the next round of potential innovation to happen in another 5-10
years after materializing the research roadmap discussed here. This research
roadmap needs a lot of research work and innovations from the community to
achieve these goals.

##  6 Conclusion

This vision paper emphasized an important systems research direction on
offloadable and migratable microservices on disaggregated datacenter
architectures. We performed a systematic survey on related studies, presented
our vision and associated research challenges, and proposed a research roadmap
to achieve the envisioned objectives. In the future, we plan to build a new
microservice or serverless computing system, which can efficiently support the
features outlined in our research agenda. We also wish our vision paper can
bring more attention from the community to collaborate and participate in this
research direction.

## Acknowledgement

We would like to thank Stanko Novakovic and anonymous reviewers for their
valuable feedback. This work was supported in part by NSF research grant CCF
#1822987.

## References

  * [1] Apache OpenWhisk.  https://openwhisk.apache.org/.
  * [2] AWS Lambda - Serverless Compute.  https://aws.amazon.com/lambda/.
  * [3] Azure Functions Serverless Architecture.  https://azure.microsoft.com/en-us/services/functions/.
  * [4] Computational SSDs.  https://www.snia.org/sites/default/files/SDCEMEA/2019/Presentations/Computational_SSDs_Final.pdf.
  * [5] Decomposing Twitter: Adventures in Service-Oriented Architecture.  https://www.slideshare.net/InfoQ/decomposing-twitter-adventures-in-serviceoriented-architecture.
  * [6] Google Cloud Functions.  https://cloud.google.com/functions/.
  * [7] Memory-Centric Active Storage.  https://ibm.github.io/mcas/.
  * [8] Microservices Workshop: Why, What, and How to Get There.  https://www.slideshare.net/adriancockcroft/microservices-workshop-craft-conference.
  * [9] The Evolution of Microservice.  https://www.slideshare.net/adriancockcroft/evolution-of-microservices-craft-conference.
  * [10] Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Stanko Novaković, Arun Ramanathan, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei.  Remote Regions: A Simple Abstraction for Remote Memory.  In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 775–787, Boston, MA, July 2018. USENIX Association.
  * [11] Krste Asanović.  FireBox: A Hardware Building Block for 2020 Warehouse-Scale Computers.  FAST ’14, Santa Clara, CA, February 2014. USENIX Association.
  * [12] Oreoluwatomiwa O. Babarinsa and Stratos Idreos.  JAFAR: Near-Data Processing for Databases.  In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD ’15, page 2069–2070, New York, NY, USA, 2015\. Association for Computing Machinery.
  * [13] Brad Calder, Ju Wang, Aaron Ogus, Niranjan Nilakantan, Arild Skjolsvold, Sam McKelvie, Yikang Xu, Shashwat Srivastav, Jiesheng Wu, Huseyin Simitci, Jaidev Haridas, Chakravarthy Uddaraju, Hemal Khatri, Andrew Edwards, Vaman Bedekar, Shane Mainali, Rafay Abbasi, Arpit Agarwal, Mian Fahim ul Haq, Muhammad Ikram ul Haq, Deepali Bhardwaj, Sowmya Dayanand, Anitha Adusumilli, Marvin McNett, Sriram Sankaran, Kavitha Manivannan, and Leonidas Rigas.  Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency.  In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP ’11, page 143–157, New York, NY, USA, 2011. Association for Computing Machinery.
  * [14] Wei Cao, Yang Liu, Zhushi Cheng, Ning Zheng, Wei Li, Wenjie Wu, Linqiang Ouyang, Peng Wang, Yijing Wang, Ray Kuan, Zhenjun Liu, Feng Zhu, and Tong Zhang.  POLARDB Meets Computational Storage: Efficiently Support Analytical Workloads in Cloud-Native Relational Database.  In 18th USENIX Conference on File and Storage Technologies (FAST 20), pages 29–41, Santa Clara, CA, February 2020. USENIX Association.
  * [15] Arup De, Maya Gokhale, Rajesh Gupta, and Steven Swanson.  Minerva: Accelerating Data Analysis in Next-Generation SSDs.  In 2013 IEEE 21st Annual International Symposium on Field-Programmable Custom Computing Machines, pages 9–16, 2013.
  * [16] Facebook.  Facebook’s New Storage Platform Design Enables Capacity to Scale Exponentially, Easily and Affordably.  https://blog.seagate.com/intelligent/facebooks-new-storage-platform-design-enables-capacity-to-scale-exponentially-easily-and-affordably/, 2020 (Accessed on 2020-12-31).
  * [17] Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, Kelvin Hu, Meghna Pancholi, Yuan He, Brett Clancy, Chris Colen, Fukang Wen, Catherine Leung, Siyuan Wang, Leon Zaruvinsky, Mateo Espinosa, Rick Lin, Zhongling Liu, Jake Padilla, and Christina Delimitrou.  An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems.  ASPLOS ’19, page 3–18, New York, NY, USA, 2019. Association for Computing Machinery.
  * [18] Saugata Ghose, Amirali Boroumand, Jeremie S. Kim, Juan Gómez-Luna, and Onur Mutlu.  Processing-In-Memory: A Workload-Driven Perspective.  IBM Journal of Research and Development, 63(6):3:1–3:19, 2019.
  * [19] Martin Grambow, Lukas Meusel, Erik Wittern, and David Bermbach.  Benchmarking Microservice Performance: A Pattern-based Approach.  In Proceedings of the 35th Annual ACM Symposium on Applied Computing, pages 232–241, 2020.
  * [20] Boncheol Gu, Andre S. Yoon, Duck-Ho Bae, Insoon Jo, Jinyoung Lee, Jonghyun Yoon, Jeong-Uk Kang, Moonsang Kwon, Chanho Yoon, Sangyeun Cho, Jaeheon Jeong, and Duckhyun Chang.  Biscuit: A Framework for Near-Data Processing of Big Data Workloads.  In Proceedings of the 43rd International Symposium on Computer Architecture, ISCA ’16, page 153–165. IEEE Press, 2016.
  * [21] Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin.  Efficient Memory Disaggregation with Infiniswap.  In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 649–667, Boston, MA, March 2017. USENIX Association.
  * [22] Shashank Gugnani, Arjun Kashyap, and Xiaoyi Lu.  Understanding the Idiosyncrasies of Real Persistent Memory.  Proceedings of the VLDB Endowment, 14(4):626–639, 2021.
  * [23] Zhiqiang He, Dongyang Wang, Binzhang Fu, Kun Tan, Bei Hua, Zhi-Li Zhang, and Kai Zheng.  MasQ: RDMA for Virtual Private Cloud.  In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM ’20, page 1–14, New York, NY, USA, 2020. Association for Computing Machinery.
  * [24] Yu-Ching Hu, Murtuza Taher Lokhandwala, Te I., and Hung-Wei Tseng.  Dynamic Multi-Resolution Data Storage.  In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’52, page 196–210, New York, NY, USA, 2019. Association for Computing Machinery.
  * [25] A.R. Hurson, L.L. Miller, S.H. Pakzad, M.H. Eich, and B. Shirazi.  Parallel Architectures for Database Systems.  volume 28 of Advances in Computers, pages 107 – 151. Elsevier, 1989\.
  * [26] Intel.  Disaggregated Servers Drive Data Center Efficiency and Innovation.  https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/disaggregated-server-architecture-drives-data-center-efficiency-paper.html, 2020 (Accessed on 2020-12-31).
  * [27] Jie Zhang and Xiaoyi Lu and Dhabaleswar K. Panda.  High-Performance Virtual Machine Migration Framework for MPI Applications on SR-IOV enabled InfiniBand Clusters.  In 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Orlando, USA, May 2017.
  * [28] Daehyeok Kim, Tianlong Yu, Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Shachar Raindel, Chuanxiong Guo, Vyas Sekar, and Srinivasan Seshan.  FreeFlow: Software-based Virtual RDMA Networking for Containerized Clouds.  In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 113–126, Boston, MA, February 2019. USENIX Association.
  * [29] Jungwon Kim, Seyong Lee, and Jeffrey S. Vetter.  PapyrusKV: A High-Performance Parallel Key-Value Store for Distributed NVM Architectures.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’17, New York, NY, USA, 2017. Association for Computing Machinery.
  * [30] Nane Kratzke and Peter-Christian Quint.  A Visualizing Network Benchmark for Microservices.  In Proceedings of the 6th International Conference on Cloud Computing and Services Science (CLOSER), 2016.
  * [31] Sergey Legtchenko, Hugh Williams, Kaveh Razavi, Austin Donnelly, Richard Black, Andrew Douglas, Nathanaël Cheriere, Daniel Fryer, Kai Mast, Angela Demke Brown, Ana Klimovic, Andy Slowey, and Antony Rowstron.  Understanding Rack-Scale Disaggregated Storage.  In Proceedings of the 9th USENIX Conference on Hot Topics in Storage and File Systems, HotStorage’17, page 2, USA, 2017. USENIX Association.
  * [32] Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K. Reinhardt, and Thomas F. Wenisch.  Disaggregated Memory for Expansion and Sharing in Blade Servers.  SIGARCH Comput. Archit. News, 37(3):267–278, June 2009.
  * [33] Ming Liu, Simon Peter, Arvind Krishnamurthy, and Phitchaya Mangpo Phothilimthana.  E3: Energy-Efficient Microservices on SmartNIC-Accelerated Servers.  In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 363–378, Renton, WA, July 2019. USENIX Association.
  * [34] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A Distributed Framework for Emerging AI Applications.  In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI’18, page 561–577, USA, 2018. USENIX Association.
  * [35] Stanko Novakovic, Alexandros Daglis, Edouard Bugnion, Babak Falsafi, and Boris Grot.  Scale-out NUMA.  In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’14, page 3–18, New York, NY, USA, 2014. Association for Computing Machinery.
  * [36] Kwanghyun Park, Yang-Suk Kee, Jignesh M Patel, Jaeyoung Do, Chanik Park, and David J Dewitt.  Query Processing on Smart SSDs.  IEEE Data Eng. Bull., 37(2):19–26, 2014.
  * [37] PCI-SIG.  Single Root I/O Virtualization.  https://pcisig.com/specifications/iov/singleroot/.
  * [38] Nathan Pemberton.  Exploring the Disaggregated Memory Interface Design Space.  http://word19.ece.cornell.edu/word19-paper8-camera.pdf.
  * [39] Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Ionnis Koltsidas, and Thomas R. Gross.  A Hybrid I/O Virtualization Framework for RDMA-Capable Network Interfaces.  In Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE ’15, page 17–30, New York, NY, USA, 2015. Association for Computing Machinery.
  * [40] Maksym Planeta, Jan Bierbaum, Leo Sahaya Daphne Antony, Torsten Hoefler, and Hermann Härtig.  MigrOS: Transparent Operating Systems Live Migration Support for Containerised RDMA-applications.  https://arxiv.org/abs/2009.06988, 2020.
  * [41] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang.  LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 69–87, Carlsbad, CA, October 2018. USENIX Association.
  * [42] Vishal Shrivastav, Asaf Valadarsky, Hitesh Ballani, Paolo Costa, Ki Suh Lee, Han Wang, Rachit Agarwal, and Hakim Weatherspoon.  Shoal: A Network Architecture for Disaggregated Racks.  In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 255–270, Boston, MA, February 2019. USENIX Association.
  * [43] Akshitha Sriraman and Thomas F. Wenisch.  µSuite: A Benchmark Suite for Microservices.  In 2018 IEEE International Symposium on Workload Characterization (IISWC), pages 1–12, 2018.
  * [44] Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang.  Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores.  In 2020 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 20), pages 33–48, 2020.
  * [45] Shin-Yeh Tsai and Yiying Zhang.  LITE Kernel RDMA Support for Datacenter Applications.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, page 306–324, New York, NY, USA, 2017. Association for Computing Machinery.
  * [46] Takanori Ueda, Takuya Nakaike, and Moriyoshi Ohara.  Workload Characterization for Microservices.  In 2016 IEEE International Symposium on Workload Characterization (IISWC), pages 1–10, 2016.
  * [47] Tobias Vinçon, Arthur Bernhardt, Ilia Petrov, Lukas Weber, and Andreas Koch.  NKV: Near-Data Processing with KV-Stores on Native Computational Storage.  In Proceedings of the 16th International Workshop on Data Management on New Hardware, DaMoN ’20, New York, NY, USA, 2020. Association for Computing Machinery.
  * [48] Dongyang Wang, Binzhang Fu, Gang Lu, Kun Tan, and Bei Hua.  VSocket: Virtual Socket Interface for RDMA in Public Clouds.  In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE 2019, page 179–192, New York, NY, USA, 2019. Association for Computing Machinery.
  * [49] Mark Wilkening, Udit Gupta, Samuel Hsia, Caroline Trippel, Carole-Jean Wu, David Brooks, and Gu-Yeon Wei.  RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inference, 2021.
  * [50] Jian Yang, Juno Kim, Morteza Hoseinzadeh, Joseph Izraelevitz, and Steve Swanson.  An Empirical Guide to the Behavior and Use of Scalable Persistent Memory.  In 18th USENIX Conference on File and Storage Technologies (FAST 20), pages 169–182, Santa Clara, CA, February 2020. USENIX Association.
  * [51] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chenjie Xu, Chao Ji, and Wenyun Zhao.  Benchmarking Microservice Systems for Software Engineering Research.  In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE ’18, page 323–324, New York, NY, USA, 2018. Association for Computing Machinery.


# Inverse Constrained Reinforcement Learning

Usman Anwar  Shehryar Malik  Alireza Aghasi  Ali Ahmed

###### Abstract

In real world settings, numerous constraints are present which are hard to
specify mathematically. However, for the real world deployment of
reinforcement learning (RL), it is critical that RL agents are aware of these
constraints, so that they can act safely. In this work, we consider the
problem of learning constraints from demonstrations of a constraint-abiding
agent’s behavior. We experimentally validate our approach and show that our
framework can successfully learn the most likely constraints that the agent
respects. We further show that these learned constraints are transferable to
new agents that may have different morphologies and/or reward functions.
Previous works in this regard have either mainly been restricted to tabular
(discrete) settings, specific types of constraints or assume the environment’s
transition dynamics. In contrast, our framework is able to learn arbitrary
Markovian constraints in high-dimensions in a completely model-free setting.
The code can be found it: https://github.com/shehryar-malik/icrl.

Machine Learning, ICML

##  1 Introduction

Reward functions are a critical component in reinforcement learning settings.
As such, it is important that reward functions are designed accurately and are
well-aligned with the intentions of the human designer. This is known as agent
(or value) alignment (see, e.g., Leike et al. (2018, 2017); Amodei et al.
(2016)). Misspecified rewards can lead to unwanted and unsafe situations (see,
e.g, Amodei & Clark (2016)). However, designing accurate reward functions
remains a challenging task. Human designers, for example, tend to prefer
simple reward functions that agree well with their intuition and are easily
interpretable. For example, a human designer might choose a reward function
that encourages an RL agent driving a car to minimize its traveling time to a
certain destination. Clearly, such a reward function makes sense in the case
of a human driver since inter-human communication is contextualized within a
framework of unwritten and unspoken constraints, often colloquially termed as
‘common-sense’. That is, while a human driver will try to minimize their
traveling time, they will be careful not to break traffic rules, take actions
that endanger passersby, and so on. However, we cannot assume such behaviors
from RL agents since they are are not imbued with common-sense constraints.

(a) Expert policy

(b) Nominal policy

(c) Recovered policy

Figure 1: A simple visualization. The agent’s reward is proportional to how
close it is to the goal at the bottom-left corner. However, the real world
(right) has a lion occupying the lower bridge. The expert is aware of this and
so takes the upper bridge. However, the lion is not present in the simulator
(middle and left). In a bid to maximize its reward the nominal policy chooses
to take the lower bridge. Clearly, such a policy will result in the agent
being eaten up by the lion in the real world. Our method, on the other hand,
tries to reconcile the reward function in the simulator with the expert’s
behavior, and ultimately learns to avoid the lower bridge.

Constrained reinforcement learning provides a natural framework for maximizing
a reward function subject to some constraints (we refer the reader to Ray et
al. (2019) for a brief overview of the field). However, in many cases, these
constraints are hard to specify explicitly in the form of mathematical
functions. One way to address this issue is to automatically extract
constraints by observing the behavior of a constraint-abiding agent. Consider,
for example, the cartoon in Figure 1. Agents start at the bottom-left corner
and are rewarded according to how quickly they reach the goal at the bottom-
right corner. However, what this reward scheme misses out is that in the real
world the lower bridge is occupied by a lion which attacks any agents
attempting to pass through it. Therefore, agents that are naïvely trained to
maximize the reward function will end up performing poorly in the real world.
If, on the other hand, the agent had observed that the expert (in the leftmost
figure) actually performed suboptimally with respect to the stipulated reward
scheme by taking a longer route to the goal, it could have concluded that (for
some unknown reason) the lower bridge must be avoided and consequently would
have not been eaten by the lion!

Scobee & Sastry (2020) formalizes this intuition by casting the problem of
recovering constraints in the maximum entropy framework for inverse RL (IRL)
(Ziebart et al., 2008) and proposes a greedy algorithm to infer the smallest
number of constraints that best explain the expert behavior. However, Scobee &
Sastry (2020) has two major limitations: it assumes (1) tabular (discrete)
settings, and (2) the environment’s transition dynamics. In this work, we aim
to address both of these issues by learning a constraint function instead
through a sample-based approximation of the objective function of Scobee &
Sastry (2020). Consequently, our approach is model-free, admits continuous
states and actions and can learn arbitrary Markovian constraints111Markovian
constraints are of the form $c(\tau)=\prod_{t=1}^{T}c(s_{t},a_{t})$, i.e., the
constraint function $c$ is independent of the past states and actions in a
trajectory. (The notation used here is introduced in the next section.).
Furthermore, we empirically show that our method scales well to high-
dimensions.

Typical inverse RL methods only make use of expert demonstrations and do not
assume any knowledge about the reward function at all. However, most reward
functions can be expressed in the form “do this task while not doing these
other things” where other things are generally constraints that a designer
wants to impose on an RL agent. The main task (“do this”) is often quite easy
to encode in the form of a simple nominal reward function. In this work, we
focus on learning the constraint part (“do not do that”) from provided expert
demonstrations and using it in conjunction with the nominal reward function to
train RL agents. In this perspective, our work can be seen as a principled way
to inculcate prior knowledge about the agent’s task in IRL. This is a key
advantage over other IRL methods which also often end up making assumptions
about the agent’s task in the form of regularizers such as in Finn et al.
(2016).

The main contribution of this work is that it formulates the problem of
inferring constraints from a set of expert demonstrations as a learning
problem which allows it to be used in continuous, high-dimensional settings.
To the best of our knowledge, this is the first work in this regard. This also
eliminates the need to assume, as Scobee & Sastry (2020) does, the
environment’s transition dynamics. Finally, we demonstrate the ability of our
method to train constraint-abiding agents in high-dimensions and show that it
can also be used to prevent reward hacking.

##  2 Preliminaries

###  2.1 Unconstrained RL

A finite-horizon Markov Decision Process (MDP) $\mathcal{M}$ is a tuple
$(\mathcal{S},\mathcal{A},p,r,\gamma,T)$, where
$\mathcal{S}\in\mathbb{R}^{|\mathcal{S}|}$ is a set of states,
$\mathcal{A}\in\mathbb{R}^{|\mathcal{A}|}$ is a set of actions,
$p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$ is the
transition probability function (where $p(s^{\prime}|s,a)$ denotes the
probability of transitioning to state $s^{\prime}$ from state $s$ by taking
action $a$), $r:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward
function, $\gamma$ is the discount factor and $T$ is the time-horizon. A
trajectory $\tau=\\{s_{1},a_{1},\ldots,s_{T},a_{T}\\}$ denotes a sequence of
states-action pairs such that $s_{t+1}\sim p(\cdot|s_{t},a_{t})$. A policy
$\pi:\mathcal{S}\mapsto\mathcal{P}(\mathcal{A})$ is a map from states to
probability distributions over actions, with $\pi(a|s)$ denoting the
probability of taking action $a$ in state $s$. We will sometimes abuse
notation to write $\pi(s,a)$ to mean the joint probability of visiting state
$s$ and taking action $a$ under the policy $\pi$ and similarly $\pi(\tau)$ to
mean the probability of the trajectory $\tau$ under the policy $\pi$.

Define $r(\tau)=\sum_{t=1}^{T}\gamma^{t}r(s_{t},a_{t})$ to be the total
discounted reward of a trajectory. Forward RL algorithms try to find an
optimal policy $\pi^{*}$ that maximizes the expected total discounted reward
$J(\pi)=\mathbb{E}_{\tau\sim\pi}[r(\tau)]$. On the other hand, given a set of
trajectories sampled from the optimal (also referred to as expert) policy
$\pi^{*}$, inverse RL (IRL) algorithms aim to recover the reward function $r$,
which can then be used to learn the optimal policy $\pi^{*}$ via some forward
RL algorithm.

###  2.2 Constrained RL

While normal (unconstrained) RL tries to find a policy that maximizes
$J(\pi)$, constrained RL instead focuses on finding a policy that maximizes
$J(\pi)$ while respecting explicitly-defined constraints. A popular framework
in this regard is the one presented in Altman (1999) which introduces the
notion of a constrained MDP (CMDP). A CMDP $\mathcal{M}^{c}$ is a simple MDP
augmented with a cost function
$c:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ and a budget $\alpha\geq 0$.
Define $c(\tau)=\sum_{t=1}^{T}\gamma^{t}c(s_{t},a_{t})$ to be the total
discounted cost of the trajectory $\tau$ and
$J^{c}(\pi)=\mathbb{E}_{\tau\sim\pi}[c(\tau)]$ to be the expected total
discounted cost. The forward constrained RL problem is to find the policy
$\pi_{c}^{*}$ that maximizes $J(\pi)$ subject to $J^{c}(\pi)\leq\alpha$.

In this work, given a set $\mathcal{D}$ of trajectories sampled from
$\pi_{c}^{*}$, the corresponding unconstrained MDP $\mathcal{M}$ (i.e.,
$\mathcal{M}^{c}$ without the cost function $c$) and a budget $\alpha$, we are
interested in recovering a cost function which when augmented with
$\mathcal{M}$ has an optimal policy that generates the same set of
trajectories as in $\mathcal{D}$. We call this as the inverse constrained
reinforcement learning (ICRL) problem.

If the budget $\alpha$ is strictly greater than $0$, then the cost function
$c$ defines soft constraints over all possible state-action pairs. In other
words, a policy is allowed to visit states and take actions that have non-zero
costs as long as the expected total discounted cost remains less than
$\alpha$. If, however, $\alpha$ is $0$ then the cost function translates into
hard constraints over all state-action pairs that have a non-zero cost
associated with them. A policy can thus never visit these state-action pairs.
In this work, we restrict ourselves to this hard constraint setting. Note that
this is not particularly restrictive since, for example, safety constraints
are often hard constraints as well are constraints imposed by physical laws.

Since we restrict ourselves to hard constraints, we can rewrite the
constrained RL problems as follows: define $\mathcal{C}=\\{(s,a)|c(s,a)\neq
0\\}$ to be the constraint set induced by $c$. The forward constrained RL
problem is to find the optimal constrained policy $\pi^{*}_{\mathcal{C}}$ that
maximizes $J(\pi)$ subject to
$\pi^{*}_{\mathcal{C}}(s,a)=0\;\forall(s,a)\in\mathcal{C}$. The inverse
constrained RL problem is to recover the constraint set $\mathcal{C}$ from
trajectories sampled from $\pi^{*}_{\mathcal{C}}$.

Finally, we will refer to our unconstrained MDP as the nominal MDP
hereinafter. The nominal MDP represents the nominal environment (simulator) in
which we train our agent.

##  3 Formulation

###  3.1 Maximum Likelihood Constraint Inference

We take Scobee & Sastry as our starting point. Suppose that we have a set of
trajectories $\mathcal{D}=\\{\mathcal{\tau}^{(i)}\\}_{i=1}^{N}$ sampled from
an expert $\pi_{\mathcal{C}}^{*}$ navigating in a constrained MDP
$\mathcal{M}^{\mathcal{C}^{*}}$ where $\mathcal{C}^{*}$ denotes the (true)
constraint set. Furthermore, we are also given the corresponding nominal MDP
$\mathcal{M}$. Our goal is to recover a constraint set which when augmented
with $\mathcal{M}$ results in a CMDP that has an optimal policy that respects
the same set of constraints as $\pi_{\mathcal{C}}^{*}$ does. Scobee & Sastry
pose this as a maximum likelihood problem. That is, if we let
$p_{\mathcal{M}}$ denote probabilities given that we are considering MDP
$\mathcal{M}$ and assume a uniform prior on all constraint sets, then we can
choose $\mathcal{C}^{*}$ according to

$\mathcal{C}^{*}\leftarrow\operatorname*{arg\,max}_{\mathcal{C}}p_{\mathcal{M}}(\mathcal{D}|\mathcal{C}).$
(1)

Under the maximum entropy (MaxEnt) model presented in Ziebart et al. (2008),
the probability of a trajectory under a deterministic MDP $\mathcal{M}$ can be
modelled as

$\pi_{\mathcal{M}}(\tau)=\frac{\exp(\beta
r(\tau))}{Z_{\mathcal{M}}}\mathbbm{1}^{\mathcal{M}}(\tau),$ (2)

where $Z_{\mathcal{M}}=\int\exp(\beta
r(\tau))\mathbbm{1}^{\mathcal{M}}(\tau)d\tau$ is the partition function,
$\beta\in[0,\infty)$ is a parameter describing how close the agent is to the
optimal distribution (as $\beta\rightarrow\infty$ the agent becomes a perfect
optimizer and as $\beta\rightarrow 0$ the agent simply takes random actions)
and $\mathbbm{1}$ is an indicator function that is $1$ for trajectories
feasible under the MDP $\mathcal{M}$ and $0$ otherwise.

Assume that all trajectories in $\mathcal{D}$ are i.i.d. and sampled from the
MaxEnt distribution. We have

$p(\mathcal{D}|\mathcal{C})=\frac{1}{\left(Z_{\mathcal{M}^{\mathcal{C}}}\right)^{N}}\prod_{i=1}^{N}\exp(\beta
r(\tau^{(i)}))\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau^{(i)}).$ (3)

Note that $\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau^{(i)})$ is $0$ for all
trajectories that contain any state-action pair that belongs to $\mathcal{C}$.
To maximize this, Scobee & Sastry propose a greedy strategy wherein they start
with an empty constraint set and incrementally add state-action pairs that
result in the maximal increase in $p(\mathcal{D}|\mathcal{C})$.

###  3.2 Sample-Based Approximation

Since we are interested in more realistic settings where the state and action
spaces can be continuous, considering all possible state-action pairs
individually usually becomes intractable. Instead, we propose a learning-based
approach wherein we try to approximate
$\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau)$ using a neural network.
Consider the log likelihood

$\begin{split}\mathcal{L(C)}&=\frac{1}{N}\log p(\mathcal{D}|\mathcal{C})\\\
&=\frac{1}{N}\sum_{i=1}^{N}\left[\beta
r(\tau^{(i)})+\log\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau^{(i)})\right]-\log
Z_{\mathcal{M}^{\mathcal{C}}}\\\ &=\frac{1}{N}\sum_{i=1}^{N}\left[\beta
r(\tau^{(i)})+\log\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau^{(i)})\right]-\\\
&\qquad\qquad\qquad\qquad\log\int\exp(\beta
r(\tau))\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau)d\tau.\end{split}$ (4)

Note that
$\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(\tau)=\prod_{t=0}^{T}\mathbbm{1}^{\mathcal{M}^{\mathcal{C}}}(s_{t},a_{t})$
merely tells us whether the trajectory $\tau$ is feasible under the constraint
set $\mathcal{C}$ or not. Let us have a binary classifier $\zeta_{\theta}$
parameterized with $\theta$ do this for us instead, i.e., we want
$\zeta_{\theta}(s_{t},a_{t})$ to be $1$ if $(s_{t},a_{t})$ is not in
$\mathcal{C}^{*}$ and $0$ otherwise. Using $\zeta_{\theta}(\tau)$ as a short
hand for $\prod_{t=0}^{T}\zeta_{\theta}(s_{t},a_{t})$, we have

$\begin{split}\mathcal{L}(\mathcal{C})&=\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\left[\beta
r(\tau^{(i)})+\log\zeta_{\theta}(\tau^{(i)})\right]\\\
&\qquad\qquad\qquad\qquad-\log\int\exp(\beta
r(\tau))\zeta_{\theta}(\tau)d\tau.\end{split}$ (5)

Let $\mathcal{M}^{\bar{\zeta}_{\theta}}$ denote the MDP obtained after
augmenting $\mathcal{M}$ with the cost function
$\bar{\zeta}_{\theta}:=1-\zeta_{\theta}$222Note that since we are assuming
that $\alpha$ is $0$, we can assign any non-zero (positive) cost to state-
action pairs that we want to constrain. Here $1-\zeta_{\theta}$ assigns a cost
of $1$ to all such pairs., and $\pi_{\mathcal{M}^{\bar{\zeta}_{\theta}}}$
denote the corresponding MaxEnt policy. Taking gradients of (5) with respect
to $\theta$ gives us (see Section 8.1 in the supplementary material for
derivation)

$\nabla_{\theta}\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})-\mathbb{E}_{\tau\sim\pi_{\mathcal{M}^{\bar{\zeta}_{\theta}}}}\left[\nabla_{\theta}\log\zeta_{\theta}(\tau)\right].$
(6)

Using a sample-based approximation for the right-hand term we can rewrite the
gradient as

$\nabla_{\theta}\mathcal{L}(\theta)\approx\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})-\frac{1}{M}\sum_{j=1}^{M}\nabla_{\theta}\log\zeta_{\theta}(\hat{\tau}^{(j)}),$
(7)

where $\hat{\tau}$ are sampled from $\pi_{\mathcal{M}^{\bar{\zeta}_{\theta}}}$
(discussed in the next section). Notice that making
$\nabla_{\theta}\mathcal{L}(\theta)$ zero essentially requires matching the
expected gradient of $\log\zeta_{\theta}$ under the expert (left hand term)
and nominal (right hand term) trajectories. For brevity, we will write
$\pi_{\mathcal{M}^{\bar{\zeta}_{\theta}}}$ as $\pi_{\theta}$ from now onwards.
We can choose $\zeta_{\theta}$ to be a neural network with parameters $\theta$
and a sigmoid at the output. We train our neural network via gradient descent
by using the expression for the gradient given above.

In practice, since we have a limited amount of data, $\zeta_{\theta}$,
parameterized as a neural network, will tend to overfit. To mitigate this, we
incorporate the following regularizer into our objective function.

$R(\theta)=-\delta\sum_{\tau\sim\\{\mathcal{D},\mathcal{S}\\}}\left|1-\zeta_{\theta}(\tau)\right|$
(8)

where $\mathcal{S}$ denotes the set of trajectories sampled from
$\pi_{\theta}$ and $\delta\in[0,1)$ is a fixed constant. $R$ incentivizes
$\zeta_{\theta}$ to predict values close to $1$ (recall that
$\zeta_{\theta}\in(0,1)$), thus encouraging it to choose the smallest number
of constraints that best explain the expert data.

###  3.3 Forward Step

To evaluate (7) we need to sample from $\pi_{\theta}$. Recall that
$\pi_{\theta}$ needs to maximize $J(\pi)$ subject to
$\pi_{\theta}(s,a)=0\;\forall(s,a)\in\mathcal{Z}$ where $\mathcal{Z}$ is the
constraint set induced by $\bar{\zeta}_{\theta}$. However, since
$\zeta_{\theta}$ outputs continuous values in the range $(0,1)$, we instead
solve the soft constrained RL version, wherein we try to find a policy $\pi$
that maximizes $J(\pi)$ subject to
$\mathbb{E}_{\tau\sim\pi}[\bar{\zeta}_{\theta}(\tau)]\leq\alpha$. In our
experiments, we set $\alpha$ to a very small value. Note that if $\alpha$ is
strictly set to $0$ our optimization program will have an empty feasible set.

We represent our policy as a neural network with parameters $\phi$ and train
it through the constrained policy optimization algorithm introduced in Tessler
et al. (2019), which essentially tries to solves the following equivalent
unconstrained min-max problem on the Lagrangian of the objective function

$\min_{\lambda\geq
0}\max_{\phi}J(\pi^{\phi})+\frac{1}{\beta}\mathcal{H}(\pi^{\phi})-\lambda(\mathbb{E}_{\tau\sim\pi^{\phi}}[\bar{\zeta}_{\theta}(\tau)]-\alpha)$
(9)

by gradient ascent on $\phi$ (via the Proximal Policy Optimization (PPO)
algorithm (Schulman et al., 2017)) and gradient descent on the Lagrange
multiplier $\lambda$. Note that we also add the entropy
$\mathcal{H}(\pi^{\phi})=-\mathbb{E}_{\tau\sim\pi^{\phi}}[\log\pi^{\phi}(\tau)]$
of $\pi^{\phi}$ to our objective function. Maximizing the entropy ensures that
we recover the MaxEnt distribution given in (2) at convergence (see Section
8.4 in the supplementary material for proof).

###  3.4 Incorporating Importance Sampling

Running the forward step in each iteration is typically very time consuming.
To mitigate this problem, instead of approximating the expectation in (6) with
samples from $\pi_{\theta}$, we approximate it with samples from an older
policy $\pi_{\bar{\theta}}$ where $\bar{\theta}$ were the parameters of
$\zeta$ at some previous iteration. We, therefore, only need to learn a new
policy after a fixed number of iterations. To correct for the bias introduced
in the approximation because of using a different sampling distribution, we
add importance sampling weights to our expression for the gradient. In this
case, the per step importance sampling weights can be shown to be given by
(see Section 8.2 in the supplementary material for derivation)

$\omega(s_{t},a_{t})=\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}.$
(10)

Note that the importance sampling weights at each timestep only depend on the
state and action at that timestep (and not the past and/or future states and
actions). The gradient can thus be rewritten as

$\begin{split}\nabla_{\theta}\mathcal{L}(\theta)&\approx\frac{1}{N}\sum_{i=1}^{N}\sum_{s_{t},a_{t}\in\tau^{(i)}}\nabla_{\theta}\log\zeta_{\theta}(s_{t},a_{t})-\\\
&\qquad\frac{1}{M}\sum_{j=1}^{M}\sum_{\hat{s}_{t},\hat{a}_{t}\in\tau^{(j)}}\omega(\hat{s}_{t},\hat{a}_{t})\nabla_{\theta}\log\zeta_{\theta}(\hat{s}_{t},\hat{a}_{t}),\end{split}$
(11)

where $\\{\tau^{(j)}\\}_{j=1}^{M}$ are sampled from $\pi_{\bar{\theta}}$.

(a) LapGridWorld

(b) HalfCheetah

(c) Ant

(d) Point

(e) AntBroken

Figure 2: The environments used in the experiments. Note that nominal agents
are not aware of the constraints shown.

###  3.5 KL-Based Early Stopping

Importance sampling allows us to run multiple updates of $\zeta_{\theta}$
after each forward step. However, importance sampling weights may have high
variance, which, in practice, may result in bad updates to $\zeta_{\theta}$
especially when policies $\pi_{\bar{\theta}}$ and $\pi_{\theta}$ become very
“different”. To mitigate this, we impose an upper bound on the forward and
reverse KL divergences between $\pi_{\bar{\theta}}$ and $\pi_{\theta}$. We do
this by terminating updates of $\zeta_{\theta}$ once the KL-divergences exceed
some pre-defined threshold. The forward and reverse KL-divergences can be
bounded in terms of the importance sampling weights as follows (see Section
8.3 in the supplementary material for derivation).

$\begin{split}D_{\text{KL}}(\pi_{\bar{\theta}}||\pi_{\theta})&\leq
2\log\bar{\omega}\\\
D_{KL}(\pi_{\theta}||\pi_{\bar{\theta}})&\leq\frac{\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[(\omega(\tau)-\bar{\omega})\log\omega(\tau)\right]}{\bar{\omega}}.\end{split}$
(12)

Here
$\bar{\omega}=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]$
and $\omega(\tau)=\prod_{t=1}^{T}\omega(s_{t},a_{t})$.

Algorithm 1 summarizes our training procedure.

Algorithm 1 ICRL

Input: Expert trajectories $\mathcal{D}$, iterations $N$, backward iterations
$B$, maximum allowable KLs $\epsilon_{F}$ and $\epsilon_{R}$

Initialize $\theta$ and $\phi$ randomly

for $i=1$ to $N$ do

Learn $\pi^{\phi}$ by solving (9) using current $\zeta_{\theta}$

for $j=1$ to $B$ do

Sample set of trajectories $\mathcal{S}=\\{\tau^{(k)}\\}_{k=1}^{M}$ from
$\pi^{\phi}$

Compute importance sampling weights $w(\tau^{(k)})$ using (10) for
$k=1,\ldots,M$

Use $\mathcal{S}$ and $\mathcal{D}$ to update $\theta$ via SGD by using the
gradient in (11)

Compute forward and reverse KLs using (12)

if forward KL $\geq\epsilon_{F}$ or reverse KL $\geq\epsilon_{R}$: then

break

end if

end for

end for

Reward (higher is better):

Constraint violations (lower is better):

(a) LapGridWorld

(b) HalfCheetah

(c) Ant

Figure 3: Performance of agents during training over several seeds (5 in
LapGridWorld, 10 in others). The x-axis is the number of timesteps taken in
the environment. The shaded regions correspond to the standard error.

(a) Point

(b) Ant-Broken

Figure 4: Transferring constraints. The x-axis is the number of timesteps
taken in the environment. All plots were smoothed and averaged over 5 seeds.
The shaded regions correspond to the standard error.

##  4 Experiments

We conduct experiments to validate our method and assess the quality of
constraints it recovers. In the following discussion, we will refer to both
the nominal and constrained environments. The nominal environment corresponds
to the simulator we use to train our agent (also called as the nominal agent).
It does not include any information about the constraints. The constrained (or
true) environment refers to the real-world setting where episodes are
terminated if the agent violates any constraints. We report two metrics for
each experiment that we conduct: (1) the true reward which corresponds to the
reward accumulated by the agent in the constrained environment, and (2) the
average number of constraint violations per each timestep. For the latter
metric, we assume that the true constraints are known. However, the training
algorithm does not require this information. Note that since episodes get
terminated in the constrained environment as soon as an agent violates some
constraint, agents simply trained to maximize the reward in the nominal
environment will only be able to accumulate a small amount of reward in the
constrained environment.

Furthermore, we construct the following two baselines for comparisons:

  * •

Binary Classifier (BC): We simply train a binary classifier (using the cross-
entropy loss) to classify between nominal and expert trajectories. We train
the agent by solving, as before, (9) (note that the binary classifier
corresponds to $\zeta$).

  * •

GAIL-Constraint (GC): This baseline is based on a well-known imitation
learning method called GAIL (Ho & Ermon, 2016). Since GAIL does not assume
knowledge of the reward function, while our method does, we construct the
following variant for a fairer comparison. We begin by defining the following
reward function: $\bar{r}(s,a):=r(s,a)+\log\zeta(s,a)$. Note that since for
constrained state-action pairs $\bar{r}$ will be $-\infty$, agents maximizing
$\bar{r}$ will try to satisfy all constraints. Now since we already know $r$,
we only parameterize $\zeta$ as a neural network which, then, corresponds to
the GAIL’s discriminator.

We design our experiments to answer the following two questions:

  * •

Can ICRL train agents to abide by the constraints that an expert does?

  * •

Are the constraints recovered by ICRL transferable to other agents with
possibly different morphologies and/or reward functions?

As we show below, the answer to both of these questions is in the affirmative.
Furthermore, while the GC baseline performs equally well to ICRL in the first
case, it completely fails in the second one.

Finally, all hyperparameters and additional details of the experiments can be
found in Section 8.5 in the supplementary material.

###  4.1 Learning Constraints

In this section, we design experiments to test whether ICRL can train agents
to abide by the constraints that an expert agent does.

Reward hacking: Reward hacking refers to situations in which agents optimize
their rewards but their behavior is misaligned with the intentions of the
users, e.g., when a vaccum cleaner ejects dust so that it can collect even
more (Russell & Norvig, 2010). We design a simple environment called
LapGridWorld to see if our method can avoid this behavior.. In this
environment333This environment is similar to the boat race environment in
Leike et al. (2017). (shown in Figure 2(a)) an agent moves on a rectangular
track. The agent is intended to sail clockwise around the track. Each time it
drives onto a golden dollar tile, it gets a reward of 3. However, the nominal
agent “cheats” by stepping back and forth on one dollar tile, rather than
going around the track, and ends up getting more reward than the expert (which
goes around the track, as intended).

High dimensions: For this experiment, we use two simulated robots called
HalfCheetah and Ant from OpenAI Gym (Brockman et al., 2016). The robots are
shown in Figures 2(b) and 2(c) respectively. The HalfCheetah robot has $18$
state and $6$ action dimensions and is rewarded proportionally to the distance
it covers in each step. The Ant robot has $113$ state and $8$ action
dimensions and is rewarded proportionally to the distance it moves away from
the origin. Since, for both agents, moving backwards is relatively easier than
moving forwards, in the constrained environments, we constrain all states for
which $x\leq-3$.

We plot the results of our experiments in Figures 3. The x-axis corresponds to
the number of timesteps the agent takes in the environment. The top row
corresponds to the reward of the agent during training when evaluated in the
constrained environment. The bottom row shows the average number of
constraints that the agent violates per timestep (when run in the nominal
environment). In addition to our method (ICRL) and the two baselines, we also
show the reward and average number of constraint violations of the nominal and
expert agents. As can be observed, both ICRL and GC perform equally well and
are able to achieve expert-level performance. However, BC performs performs
much more worse than these two.

###  4.2 Transferring constraints

In many cases, constraints are actually part of the environment and are the
same for different agents (for example, all vehicles must adhere to the same
speed limit). In such instances, it is useful to first learn the constraints
using only one agent and then transfer them onto other agents. These new
agents may have different morphologies and/or reward functions. In this
section, we design experiments to assess the quality of constraints recovered
by our method and the two baselines. To do so, we try to transfer constraints
learnt in the Ant experiment in the previous section to two new agents. The
first agent, shown in Figure 2(d), is the Point robot from OpenAI gym. The
Point robot’s reward function encourages it to move counter-clockwise in a
circle at a distance of $10$ units from the origin.444This environment is
similar to the PointCircle environment in Achiam et al. (2017). The second
agent, called Ant-Broken, is the Ant robot with two of its legs disabled
(colored in blue in Figure 2(e)).555This is based on the broken ant
environment in Eysenbach et al. (2020). To transfer constraints, we simply
solve (9) using the $\zeta$ learnt in the previous section.666Note that in
this case, $\zeta$ must only be fed a subset of the state and action spaces
that are common across the two agents. As such, for the Point experiment we
re-run experiments in the previous section and train $\zeta$ only on the
$(x,y)$ position coordinates of the Ant agent, since the rest of elements in
the state space are specific to Ant.

Figure 4 shows the results. The plots represent the same metrics as in the
previous section. Note that only agents trained to respect constraints
recovered by our method are able to achieve expert-level performance. Both BC
and GC perform even worse in terms of the reward than the nominal agent, which
is simply trained to maximize the reward in the nominal environment without
worrying about any constraints. However, note that both BC and GC are able to
achieve zero cost. This suggests that both BC and GC constrain too many state-
action pairs, thus preventing agents from performing their tasks optimally.

Reward (higher is better):

Constraint violations (lower is better):

(a) No IS and no ES

(b) No IS but ES

(c) IS but no ES

(d) IS and ES

Figure 5: Ablation studies on the HalfCheetah environment. All plots were
averaged over $5$ seeds. IS refers to importance sampling and ES to early
stopping. The x-axis corresponds to the number of timesteps the agent takes in
the environment. Shaded regions correspond to the standard error.

##  5 Ablation Studies

In this section, we discuss ablation studies to assess the importance of
importance sampling and the KL-based early stopping technique. We carry out
four sets of experiments by disabling either one or both of importance
sampling and early stopping. Additionally, we also plot results for the case
when neither of importance sampling and early stopping are disabled. In each
set of experiments, we vary the number of backward iterations, $B$ (see
Algorithm 1 and observe (as in the previous section) the true reward (on the
constrained environment) and average number of constraint violations. All four
sets of experiments are shown in Figure 5. For comparison, we also show the
best true reward and lowest number of constraint violations that our method
was able to achieve with all of its components enabled (these are the same
results as in the previous section).

From the plots, it is clear that both of importance sampling and early
stopping are important to the success of our method.

##  6 Related Work

Forward Constrained RL: Several approaches have been proposed in literature to
solve the forward constrained RL problem in the context of CMDPs (Altman,
1999). Achiam et al. (2017) analytically solves trust region policy
optimization problems at each policy update to enforce the constraints. Chow
et al. (2018) uses a Lyapunov approach and also provide theoretical
guarantees. Le et al. (2019) proposes an algorithm for cases when there are
multiple constraints. Finally, a well-known approach centers around rewriting
the constrained RL problem as an equivalent unconstrained min-max problem by
using Lagrange multipliers (Zhang et al., 2019; Tessler et al., 2019;
Bhatnagar, 2010)) (see Section 3.3 for further details).

Constraint Inference: Previous work done on inferring constraints from expert
demonstrations has either focused on either inferring specific type of
constraints such as geometric (D’Arpino & Shah, 2017; Subramani et al., 2018),
sequential (Pardowitz et al., 2005) or convex (Miryoosefi et al., 2019)
constraints or is restricted to tabular settings (Scobee & Sastry, 2020; Chou
et al., 2018) or assumes transition dynamics (Chou et al., 2020).

Preference Learning: Constraint inference also links to preference learning
which aims to extract user preferences (constraints imposed by an expert on
itself, in our case) from different sources such as ratings (Daniel et al.,
2014), comparisions (Christiano et al., 2017; Sadigh et al., 2017), human
reinforcement signals (MacGlashan et al., 2017) or the initial state of the
agent’s environment (Shah et al., 2019). Preference learning also includes
inverse RL, which aims to recover an agent’s reward function by using its
trajectories. To deal with the inherent ill-posedness of this problem, inverse
RL algorithms often incorporate regularizers (Ho & Ermon, 2016; Finn et al.,
2016) or assume a prior distribution over the reward function (Jeon et al.,
2018; Michini & How, 2012; Ramachandran & Amir, 2007).

##  7 Conclusion and Future Work

We have presented a method to learn constraints from an expert’s
demonstrations. Unlike previous works, our method both learns arbitrary
constraints and can be used in continuous settings.

While we consider our method to be an important first step towards learning
arbitrary constraints in real-world continuous settings, there is still
considerable room for improvement. For example, as is the case with Scobee &
Sastry, our formulation is also based on (2) which only holds for
deterministic MDPs. Secondly, we only consider hard constraints. Lastly, one
very interesting extension of this method can be to learn constraints only
from logs data in an offline way to facilitate safe-RL in settings where it is
difficult to even build nominal simulators such as is the case for plant
controllers.

## References

  * Achiam et al. (2017) Achiam, J., Held, D., Tamar, A., and Abbeel, P.  Constrained policy optimization.  In _International Conference on Machine Learning_ , 2017.
  * Altman (1999) Altman, E.  _Constrained Markov Decision Processes_.  Chapman and Hall, 1999.
  * Amodei & Clark (2016) Amodei, D. and Clark, J.  Faulty reward functions in the wild.  https://openai.com/blog/faulty-reward-functions/, 2016.  Accessed: 2020-07-20.
  * Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., and Mané, D.  Concrete problems in AI safety, 2016.  arXiv:1606.06565.
  * Bhatnagar (2010) Bhatnagar, S.  An actor–critic algorithm with function approximation for discounted cost constrained markov decision processes.  _Systems and Control Letters_ , 59(12):760–766, 2010.
  * Biewald (2020) Biewald, L.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  OpenAI Gym, 2016.  arXiv:1606.01540.
  * Chou et al. (2018) Chou, G., Berenson, D., and Ozay, N.  Learning constraints from demonstrations, 2018.  arXiv:1812.07084.
  * Chou et al. (2020) Chou, G., Ozay, N., and Berenson, D.  Learning constraints from locally-optimal demonstrations under cost function uncertainty.  _IEEE Robotics Autom. Lett._ , 5(2):3682–3690, 2020.
  * Chow et al. (2018) Chow, Y., Nachum, O., Duenez-Guzman, E., and Ghavamzadeh, M.  A lyapunov-based approach to safe reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Daniel et al. (2014) Daniel, C., Viering, M., Metz, J., Kroemer, O., and Peters, J.  Active reward learning.  In _Proceedings of Robotics: Science and Systems_ , 2014.
  * D’Arpino & Shah (2017) D’Arpino, C. P. and Shah, J. A.  C-learn: Learning geometric constraints from demonstrations for multi-step manipulation in shared autonomy.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017.
  * Eysenbach et al. (2020) Eysenbach, B., Asawa, S., Chaudhari, S., Salakhutdinov, R., and Levine, S.  Off-dynamics reinforcement learning: Training for transfer with domain classifiers, 2020.  arXiv:2006.13916.
  * Finn et al. (2016) Finn, C., Levine, S., and Abbeel, P.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International Conference on Machine Learning_ , 2016.
  * Ho & Ermon (2016) Ho, J. and Ermon, S.  Generative adversarial imitation learning.  In _Advances in Neural Information Processing Systems_ , 2016.
  * Jeon et al. (2018) Jeon, W., Seo, S., and Kim, K.-E.  A bayesian approach to generative adversarial imitation learning.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations, ICLR_ , 2015.
  * Le et al. (2019) Le, H., Voloshin, C., and Yue, Y.  Batch policy learning under constraints.  In _International Confernce on Machine Learning, ICML_ , 2019.
  * Leike et al. (2017) Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S.  AI safety gridworlds, 2017.  arXiv:1711.09883.
  * Leike et al. (2018) Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S.  Scalable agent alignment via reward modeling: A research direction, 2018\.  arXiv:1811.07871.
  * MacGlashan et al. (2017) MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L., Taylor, M. E., and Littman, M. L.  Interactive learning from policy-dependent human feedback.  In _International Conference on Machine Learning_ , 2017.
  * Michini & How (2012) Michini, B. and How, J. P.  Bayesian nonparametric inverse reinforcement learning.  In _Machine Learning and Knowledge Discovery in Databases_ , 2012\.
  * Miryoosefi et al. (2019) Miryoosefi, S., Brantley, K., III, H. D., Dudik, M., and Schapire, R. E.  Reinforcement learning with convex constraints.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Pardowitz et al. (2005) Pardowitz, M., Zöllner, R., and Dillmann, R.  Learning sequential constraints of tasks from user demonstrations.  In _5th IEEE-RAS International Conference on Humanoid Robots_ , 2005\.
  * Ramachandran & Amir (2007) Ramachandran, D. and Amir, E.  Bayesian inverse reinforcement learning.  In _20th International Joint Conference on Artifical Intelligence_ , 2007.
  * Ray et al. (2019) Ray, A., Achiam, J., and Amodei, D.  Benchmarking safe exploration in deep reinforcement learning, 2019.  https://cdn.openai.com/safexp-short.pdf.
  * Russell & Norvig (2010) Russell, S. and Norvig, P.  _Artificial Intelligence: A Modern Approach_.  Prentice Hall, 3 edition, 2010.
  * Sadigh et al. (2017) Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A.  Active preference-based learning of reward functions.  In _Robotics: Science and Systems XIII_ , 2017.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms, 2017.  arxiv:1707.06347.
  * Scobee & Sastry (2020) Scobee, D. R. and Sastry, S. S.  Maximum likelihood constraint inference for inverse reinforcement learning.  In _International Conference on Learning Representations_ , 2020.
  * Shah et al. (2019) Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., and Dragan, A. D.  Preferences implicit in the state of the world.  In _International Conference on Learning Representations_ , 2019.
  * Subramani et al. (2018) Subramani, G., Zinn, M., and Gleicher, M.  Inferring geometric constraints in human demonstrations.  In _2nd Conference on Robot Learning (CoRL)_ , 2018.
  * Tessler et al. (2019) Tessler, C., Mankowitz, D. J., and Mannor, S.  Reward constrained policy optimization.  In _International Conference on Learning Representations_ , 2019.
  * Zhang et al. (2019) Zhang, R., Yu, T., Shen, Y., Jin, H., and Chen, C.  Text-based interactive recommendation via constraint-augmented reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Ziebart et al. (2008) Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K.  Maximum entropy inverse reinforcement learning.  In _23rd National Conference on Artificial Intelligence (AAAI)_ , 2008\.

##  8 Appendix

This appendix should be read as a continuation of the main paper.

###  8.1 Gradient of log likelihood

The gradient of (5) is

$\begin{split}\nabla_{\theta}\mathcal{L}(\theta)&=\frac{1}{N}\sum_{i=1}^{M}\left[0+\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})\right]-\frac{1}{\int\exp(\beta
r(\tau))\zeta_{\theta}(\tau)d\tau}\int\exp(\beta
r(\tau))\nabla_{\theta}\zeta_{\theta}(\tau)d\tau\\\
&=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})-\int\frac{\exp(\beta
r(\tau))\zeta_{\theta}(\tau)}{\int\exp(\beta
r(\tau^{\prime}))\zeta_{\theta}(\tau^{\prime})d\tau^{\prime}}\nabla_{\theta}\log\zeta_{\theta}(\tau)d\tau\\\
&=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})-\int
p_{\mathcal{M}^{\bar{\zeta}_{\theta}}}(\tau)\nabla_{\theta}\log\zeta_{\theta}(\tau)d\tau\\\
&=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\log\zeta_{\theta}(\tau^{(i)})-\mathbb{E}_{\tau\sim\pi_{\mathcal{M}^{\zeta_{\theta}}}}\left[\nabla_{\theta}\log\zeta_{\theta}(\tau)\right],\end{split}$
(13)

where the second line follows from the identity
$\nabla_{\theta}\zeta_{\theta}(\tau)\equiv\zeta_{\theta}(\tau)\nabla_{\theta}\log\zeta_{\theta}(\tau)$
and the fourth line from the MaxEnt assumption.

###  8.2 Derivation of importance sampling weights

Suppose that at some iteration of our training procedure we are interested in
approximating the gradient of the log of the partition function
$\nabla_{\theta}\log Z_{\theta}$ (where $\theta$ are the current parameters of
our classifier) using an older policy $\pi_{\zeta_{\bar{\theta}}}$ (where
$\bar{\theta}$ were the parameters of the classifier which induced the
constraint set that this policy respects). We can do so by noting that

$\begin{split}Z_{\theta}&=\int\exp(r(\tau))\zeta_{\theta}(\tau)d\tau\\\
&=\int\pi_{\zeta_{\bar{\theta}}}(\tau)\left[\frac{\exp(r(\tau))\zeta_{\theta}(\tau)}{\pi_{\zeta_{\bar{\theta}}}(\tau)}\right]d\tau\\\
&=\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\exp(r(\tau))\zeta_{\theta}(\tau)}{\pi_{\zeta_{\bar{\theta}}}(\tau)}\right]\\\
&=Z_{\bar{\theta}}\cdot\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\right].\\\
\end{split}$ (14)

where the fourth lines follows from our MaxEnt assumption, i.e.,
$\pi_{\zeta_{\bar{\theta}}}(\tau)=\exp(r(\tau))\zeta_{\bar{\theta}}(\tau)/Z_{\bar{\theta}}.$

Therefore

$\begin{split}\nabla_{\theta}\log
Z_{\theta}&=\frac{1}{Z_{\theta}}\nabla_{\theta}Z_{\theta}\\\
&=\frac{1}{Z_{\bar{\theta}}\cdot\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\right]}\left[Z_{\bar{\theta}}\cdot\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\nabla_{\theta}\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\right]\right]\\\
&=\frac{1}{\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\right]}\left[\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\nabla_{\theta}\log\zeta_{\theta}(\tau)\right]\right].\end{split}$
(15)

Note that
$\mathbb{E}_{\tau\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\right]=\int\pi_{\zeta_{\theta}}(\tau)d\tau=1$.
So

$\begin{split}\nabla_{\theta}\log
Z_{\theta}&=\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(\tau)}{\zeta_{\bar{\theta}}(\tau)}\nabla_{\theta}\log\zeta_{\theta}(\tau)\right]\\\
&=\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\prod_{t=1}^{T}\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}\nabla_{\theta}\log\prod_{t^{\prime}=1}^{T}\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right]\\\
&=\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\prod_{t=1}^{T}\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}\sum_{t^{\prime}=1}^{T}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right]\\\
&=\sum_{t^{\prime}=1}^{T}\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\prod_{t=1}^{T}\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right]\\\
&=\sum_{t^{\prime}=1}^{T}\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\left(\prod_{\begin{subarray}{c}t=1\\\
t\neq
t^{\prime}\end{subarray}}^{T}\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}\right)\left(\frac{\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})}{\zeta_{\bar{\theta}}(s_{t^{\prime}},a_{t^{\prime}})}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right)\right]\\\
&=\sum_{t^{\prime}=1}^{T}\mathbb{E}_{\tau/(s_{t^{\prime}},a_{t^{\prime}})\sim\pi_{\zeta_{\bar{\theta}}}}\left[\prod_{\begin{subarray}{c}t=1\\\
t\neq
t^{\prime}\end{subarray}}^{T}\frac{\zeta_{\theta}(s_{t},a_{t})}{\zeta_{\bar{\theta}}(s_{t},a_{t})}\right]\mathbb{E}_{s_{t^{\prime}},a_{t^{\prime}}\sim\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})}{\zeta_{\bar{\theta}}(s_{t^{\prime}},a_{t^{\prime}})}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right]\\\
&=\sum_{t^{\prime}=1}^{T}\frac{Z_{\theta}}{Z_{\bar{\theta}}}\cdot\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})}{\zeta_{\bar{\theta}}(s_{t^{\prime}},a_{t^{\prime}})}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right].\\\
&\approx\sum_{t^{\prime}=1}^{T}\mathbb{E}_{\pi_{\zeta_{\bar{\theta}}}}\left[\frac{\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})}{\zeta_{\bar{\theta}}(s_{t^{\prime}},a_{t^{\prime}})}\nabla_{\theta}\log\zeta_{\theta}(s_{t^{\prime}},a_{t^{\prime}})\right],\\\
\end{split}$ (16)

where the last step assumes that $Z_{\theta}\approx Z_{\bar{\theta}}$. This is
justified since we restrict the extent to which $\zeta_{\theta}$ can change
via the early stopping technique.

###  8.3 Forward and reverse KL divergences between two policies

Consider two policies $\pi_{\bar{\theta}}$ and $\pi_{\theta}$. Using our
MaxEnt assumption, we can write the forward KL divergence as

$\begin{split}D_{KL}(\pi_{\bar{\theta}}||\pi_{\theta})&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\log\frac{\pi_{\bar{\theta}}(\tau)}{\pi_{\theta}(\tau)}\right]\\\
&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\log\frac{\zeta_{\bar{\theta}}(\tau)}{\zeta_{\theta}(\tau)}\right]+\log\frac{Z_{\theta}}{Z_{\bar{\theta}}}.\end{split}$
(17)

Let $\omega(\tau)$ denote $\zeta_{\bar{\theta}}(\tau)/\zeta_{\theta}(\tau)$.
Plugging in the expression for $Z_{\theta}$ from (14) and using Jensen’s
inequality gives

$\begin{split}D_{KL}(\pi_{\bar{\theta}}||\pi_{\theta})&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\log\omega(\tau)\right]+\log\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]\\\
&\leq
2\log\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right].\end{split}$
(18)

Similarly, the reverse KL divergence is

$\begin{split}D_{KL}(\pi_{\theta}||\pi_{\bar{\theta}})&=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\log\frac{\pi_{\theta}(\tau)}{\pi_{\bar{\theta}}(\tau)}\right]\\\
&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\frac{\pi_{\theta}(\tau)}{\pi_{\bar{\theta}}(\tau)}\log\frac{\pi_{\theta}(\tau)}{\pi_{\bar{\theta}}(\tau)}\right]\\\
&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\frac{Z_{\bar{\theta}}}{Z_{\theta}}\log\omega(\tau)\frac{Z_{\bar{\theta}}}{Z_{\theta}}\right]\\\
&=\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\log\omega(\tau)\right]\frac{Z_{\bar{\theta}}}{Z_{\theta}}+\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]\frac{Z_{\bar{\theta}}}{Z_{\theta}}\log\frac{Z_{\bar{\theta}}}{Z_{\theta}}.\\\
\end{split}$ (19)

From (14) we know that
$Z_{\bar{\theta}}/Z_{\theta}=1/\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\omega(\tau)$.
Using Jensen’s inequality we have

$\begin{split}D_{KL}(\pi_{\theta}||\pi_{\bar{\theta}})&=\frac{1}{\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]}\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\log\omega(\tau)\right]-\log\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]\\\
&\leq\frac{1}{\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\right]}\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\omega(\tau)\log\omega(\tau)\right]-\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[\log\omega(\tau)\right].\\\
\end{split}$ (20)

Letting $\bar{\omega}$ denote
$\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}[\omega(\tau)]$ gives us

$D_{KL}(\pi_{\theta}||\pi_{\bar{\theta}})\leq\frac{\mathbb{E}_{\tau\sim\pi_{\bar{\theta}}}\left[(\omega(\tau)-\bar{\omega})\log\omega(\tau)\right]}{\bar{\omega}}.$
(21)

###  8.4 Rationale for (9)

Consider a constrained MDP $\mathcal{M}^{\mathcal{C}}$ as defined in Section
2.2. We are interested in recovering the following policy

$\pi_{\mathcal{M}^{\mathcal{C}}}(\tau)=\frac{\exp(\beta
r(\tau))}{Z_{\mathcal{M}^{\mathcal{C}}}}\mathbbm{1}^{\mathcal{C}}(\tau)$ (22)

where $Z_{\mathcal{M}^{\mathcal{C}}}=\int\exp(\beta
r(\tau))\mathbbm{1}^{\mathcal{C}}(\tau)d\tau$ is the partition function and
$\mathbbm{1}^{\mathcal{C}}$ is an indicator function that is $0$ if
$\tau\in\mathcal{C}$ and $1$ otherwise.

Lemma: The Boltzmann policy $\pi^{B}(\tau)=\exp(\beta r(\tau))/Z$ maximizes
$\mathcal{L}(\pi)=\mathbb{E}_{\tau\sim\pi}[r(\tau)]+\frac{1}{\beta}\mathcal{H}(\pi)$,
where $H(\pi)$ denotes the entropy of $\pi$.

Proof: Note that the KL-divergence, $D_{KL}$, between a policy $\pi$ and
$\pi^{B}$ can be written as

$\begin{split}\mathcal{D}_{KL}(\pi||\pi^{B})&=\mathbb{E}_{\tau\sim\pi}[\log\pi(\tau)-\log\pi^{B}(\tau)]\\\
&=\mathbb{E}_{\tau\sim\pi}[\log\pi(\tau)-\beta r(\tau)+\log Z]\\\
&=-\mathbb{E}_{\tau\sim\pi}[\beta r(\tau)]-\mathcal{H}(\pi)+\log Z\\\
&=-\beta\mathcal{L}(\pi)+\log Z.\end{split}$ (23)

Since $\log Z$ is constant, minimizing $D_{KL}(\pi||\pi^{B})$ is equivalent to
maximizing $\mathcal{L}(\pi)$. Also, we know that
$\mathcal{D}_{KL}(\pi||\pi^{B})$ is minimized when $\pi=\pi^{B}$. Therefore,
$\pi^{B}$ maximizes $\mathcal{L}$.

Proposition: The policy in (22) is a solution of

$\underset{\lambda\geq
0}{\text{minimize}}\max_{\pi}\mathbb{E}_{\tau\sim\pi}[r(\tau)]+\frac{1}{\beta}\mathcal{H}(\pi^{\phi})-\lambda(\mathbb{E}_{\tau\sim\pi^{\phi}}[\bar{\zeta}_{\theta}(\tau)]-\alpha).$
(24)

Proof: Let us rewrite the inner optimization problem as

$\max_{\pi}\;\mathbb{E}_{\tau\sim\pi}[r(\tau)-\lambda(\bar{\zeta}_{\theta}(\tau)-\alpha)]+\frac{1}{\beta}\mathcal{H}(\pi).$
(25)

From the Lemma we know that the solution to this is

$\pi(\tau,\lambda)=\frac{g(\tau,\lambda)}{\int
g(\tau^{\prime},\lambda)d\tau^{\prime}},$ (26)

where
$g(\tau,\lambda)=\exp(\beta(r(\tau)-\lambda(\bar{\zeta}_{\theta}(\tau)-\alpha)))$.
To find $\pi^{*}(\tau)=\min_{\lambda}\pi(\tau,\lambda)$, note that:

  1. 1.

When $\bar{\zeta}_{\theta}(\tau)\leq\alpha$, then $\lambda^{*}=0$ minimizes
$\pi$. In this case $g(\tau,\lambda^{*})=\exp(\beta r(\tau))$.

  2. 2.

When $\bar{\zeta}_{\theta}(\tau)>\alpha$, then $\lambda^{*}\rightarrow\infty$
minimizes $\pi$. In this case $g(\tau,\lambda^{*})=0$.

We can combine both of these cases by writing

$\pi^{*}(\tau)=\frac{\exp(r(\tau))}{\int\exp(r(\tau^{\prime}))\mathbbm{1}^{\bar{\zeta}_{\theta}}(\tau^{\prime})d\tau^{\prime}}\mathbbm{1}^{\bar{\zeta}_{\theta}}(\tau),$
(27)

where $\mathbbm{1}^{\bar{\zeta}_{\theta}}(\tau)$ is $1$ if
$\bar{\zeta}_{\theta}(\tau)\leq\alpha$ and $0$ otherwise. (Note that the
denominator is greater than $0$ as long as we have at least one $\tau$ for
which $\bar{\zeta}_{\theta}(\tau)\leq\alpha$, i.e., we have at least one
feasible solution.)

###  8.5 Experimental settings

We used W&B (Biewald, 2020) to manage our experiments and conduct sweeps on
hyperparameters. We used Adam (Kingma & Ba, 2015) to optimize all of our
networks. All important hyperparameters are listed in Table 1. For the
ablation studies we used the same parameters as listed in the table for
HalfCheetah. Details on the environments can be found below.

####  8.5.1 LapGridWorld

Here, agents move on a $11\times 11$ grid by taking either clockwise or anti-
clockwise actions. The agent is awarded a reward $3$ each time it moves onto a
bridge with a dollar (see Figure 2). The agent’s state is the number of the
grid it is on.

####  8.5.2 HalfCheetah, Ant and Ant-Broken

The original reward schemes for HalfCheetah and Ant in OpenAI Gym (Brockman et
al., 2016), reward the agents proportional to the distance they cover in the
forward direction. We modify this and instead simply reward the agents
according to the amount of distance they cover (irrespective of the direction
they move in). For Ant-Broken we simply disable two of the legs of Ant by
hard-coding a torque of $0$ on their motors.

####  8.5.3 Point

For the Point agent, the reward function at each timestep is defined as
follows

$r:=\frac{ydx-xdx}{\left(1+|\sqrt{x^{2}+y^{2}}-10)|\right)}$ (28)

where $x,y$ are the position coordinates of the agent and $dx$ and $dy$ are
the distances that the agent has moved in $x$ and $y$ directions respectively
in that timestep.

Table 1: List of hyperparameters. For neural network architectures we report
the number of hidden units in each layer. All hidden layers use the tanh
activation function.
Parameter | LapGridWorld | HalfCheetah | Ant | Point | AntBroken
---|---|---|---|---|---
Policy, $\pi_{\phi}$ |  |  |  |  |
Architecture |  |  |  |  |
Policy Network | 64, 64 | 64, 64 | 64, 64 | 64, 64 | 64, 64
Value Network | 64, 64 | 64, 64 | 64, 64 | 64, 64 | 64, 64
Cost Value Network | 64, 64 | 64, 64 | 64, 64 | 64, 64 | 64, 64
Batch Size | 64 | 64 | 128 | 64 | 128
PPO Target KL | $0.01$ | 0.01 | 0.01 | 0.01 | 0.01
Learning Rate | $3\times 10^{-4}$ | $3\times 10^{-4}$ | $3\times 10^{-5}$ | $3\times 10^{-4}$ | $3\times 10^{-5}$
Reward-GAE-$\gamma$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.99
Reward-GAE-$\lambda$ | 0.95 | 0.95 | 0.90 | 0.95 | 0.90
Cost-GAE-$\gamma$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.99
Cost-GAE-$\lambda$ | 0.95 | 0.95 | 0.95 | 0.95 | 0.95
Entropy Bonus, $1/\beta$ | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Lagrangian, $\lambda$ |  |  |  |  |
Initial Value | 1.0 | 1.0 | 0.1 | 1.0 | 0.1
Learning Rate | 0.1 | 0.1 | 1.0 | 0.1 | 1.0
Budget | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Constraint Function, $\zeta_{\theta}$ |  |  |  |  |
Architecture | 20 | 20 | 40,40 | - | -
Learning Rate | 0.01 | 0.01 | 0.01 | - | -
Backward Iterations | 10 | 10 | 10 | - | -
Regularizer Weight | 0.5 | 0.5 | 0.6 | - | -
Max Forward KL, $\epsilon_{F}$ | 10 | 10 | 10 | - | -
Max Backward KL, $\epsilon_{B}$ | 2.5 | 2.5 | 2.5 | - | -
Miscellaneous |  |  |  |  |
Expert Rollouts | 1 | 10 | 45 | - | -
Rollout Length | 200 | 1000 | 500 | 150 | 500