# OpenRL: A Unified Reinforcement Learning Framework

Shiyu Huang1,†<EMAIL_ADDRESS>Wentse Chen2<EMAIL_ADDRESS>Yiwen
Sun3<EMAIL_ADDRESS>Fuqing Bie4<EMAIL_ADDRESS>Wei-Wei Tu1,†
<EMAIL_ADDRESS>
1 4Paradigm Inc., Beijing, China
2 Robotics Institute, Carnegie Mellon University, Pittsburgh, USA
3 School of Data Science, Fudan University, Shanghai, China
4 Beijing University of Posts and Telecommunications, Beijing, China
† corresponding authors

###### Abstract

We present OpenRL, an advanced reinforcement learning (RL) framework designed
to accommodate a diverse array of tasks, from single-agent challenges to
complex multi-agent systems. OpenRL’s robust support for self-play training
empowers agents to develop advanced strategies in competitive settings.
Notably, OpenRL integrates Natural Language Processing (NLP) with RL, enabling
researchers to address a combination of RL training and language-centric tasks
effectively. Leveraging PyTorch’s robust capabilities, OpenRL exemplifies
modularity and a user-centric approach. It offers a universal interface that
simplifies the user experience for beginners while maintaining the flexibility
experts require for innovation and algorithm development. This equilibrium
enhances the framework’s practicality, adaptability, and scalability,
establishing a new standard in RL research. To delve into OpenRL’s features,
we invite researchers and enthusiasts to explore our GitHub repository at
https://github.com/OpenRL-Lab/openrl and access our comprehensive
documentation at https://openrl-docs.readthedocs.io.

Keywords: Reinforcement Learning, Multi-agent Learning, Self-play, Natural
Language Processing, PyTorch

##  1 Introduction

Reinforcement learning (RL) has undergone significant expansion in recent
years (Arulkumaran et al., 2017). This rapidly evolving field has diversified
its applications, ranging from single-agent problems (Bellemare et al., 2013;
Lillicrap et al., 2015) to the complexities of multi-agent scenarios (Yu et
al., 2022; Lin et al., 2023), from mastering self-play (Berner et al., 2019)
to the subtleties of offline RL (Fu et al., 2020; Qin et al., 2022). These
advancements in RL now empower robots to navigate challenging terrains
(Polydoros and Nalpantidis, 2017; Margolis and Agrawal, 2023; Cheng et al.,
2023), enhance large language models in understanding and generating human-
like text (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022),
and aid industrial tasks through RL-driven optimization strategies Roy et al.
(2021); Degrave et al. (2022). However, this rapid development also presents
challenges, as existing RL frameworks struggle to meet the community’s growing
demands, highlighting the need for an inclusive and adaptable framework.

Table 1: Comparison between OpenRL and other popular RL libraries. Library | NLP | Multi-agent | Self-Play | Offline RL | DeepSpeed
---|---|---|---|---|---
OpenRL | $\usym{2713}$ | $\usym{2713}$ | $\usym{2713}$ | $\usym{2713}$ | $\usym{2713}$
SB3 | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$
Ray/RLlib | $\usym{2717}$ | $\usym{2713}$ | $\usym{2713}$ | $\usym{2713}$ | $\usym{2717}$
DI-engine | $\usym{2717}$ | $\usym{2713}$ |  | $\usym{2713}$ | $\usym{2717}$
Tianshou | $\usym{2717}$ |  |  | $\usym{2713}$ | $\usym{2717}$
MARLlib | $\usym{2717}$ | $\usym{2713}$ |  | $\usym{2717}$ | $\usym{2717}$
RL4LMs | $\usym{2713}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$
trl | $\usym{2713}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2713}$
TimeChamber | $\usym{2717}$ | $\usym{2717}$ | $\usym{2717}$ | $\usym{2713}$ | $\usym{2717}$

In response to this, we introduce OpenRL, a comprehensive framework designed
for RL research and development. OpenRL aims to address key challenges in the
field: (1) Comprehensive Integration: OpenRL embraces a wide spectrum of RL
scenarios, including single-agent tasks, multi-agent interactions, and offline
RL nuances. Furthermore, OpenRL stands out in two critical emerging areas: the
realm of self-play and the convergence of NLP with RL, enabling cross-
disciplinary research through integrated tools and resources. (2) Modularity
and User-centric Design: Built on PyTorch (Paszke et al., 2019), OpenRL
features a scalable design with an intuitive interface that melds complexity
with ease of use. Our Gallery module showcases a set of ”reproducibility
scripts” for various algorithms and environments. These scripts give novices a
clear glimpse into OpenRL’s capabilities while serving as a launchpad for
experienced users. Complementing these are our detailed documentation and
tutorials, ensuring a smooth transition from understanding core concepts to
practical implementation.

Figure 1: The OpenRL framework is a three-tiered architecture. The
_Encapsulation_ layer, packages external libraries and essential components
required for subsequent modules. The _Component_ layer establishes core
interfaces for user interaction, including three high-level modules:
_Environment_ , _Network_ , and _Agent_. These modules are instrumental across
different tasks and algorithms, sharing a consistent calling interface. The
_Tool_ layer equips users with multiple utilities to facilitate training,
testing, and visualization. OpenRL enables customization of environments,
algorithms, and visualizations, offering a versatile and efficient
reinforcement learning development framework.

##  2 Design of OpenRL

OpenRL is an open-source reinforcement learning framework that features a
simple, universal interface, supporting training across a wide range of tasks
and environments. Figure 1 illustrates the overall architecture of OpenRL. The
same interface is used for all tasks, including offline reinforcement
learning. We present OpenRL’s general code interface in Appendix A. This
unified interface simplifies the process of setting up and running
reinforcement learning tasks, making OpenRL accessible to both beginners and
experienced researchers.

###  2.1 Modularity Design

Modularity is a key aspect of OpenRL’s design, contributing significantly to
its extensibility. We have abstracted various components of the framework into
distinct modules, including the _Reward Module_ , _Network Module_ ,
_Algorithm Module_ , _Environment Module_ , _Agent Module_ , and _Buffer
Module_. Due to space constraints, we will focus on three main modules: the
_Reward Module_ , _Network Module_ , and _Algorithm Module_.

Reward Module: Modifying rewards is a common practice in diverse reinforcement
learning algorithms (Eysenbach et al., 2018; Chen et al., 2022), exploration
algorithms (Burda et al., 2018; Zhang et al., 2020), and RLHF/RLAIF (RL from
AI feedback) (Ouyang et al., 2022; Bai et al., 2022). We provide a base class
for the _Reward Module_ , which users can extend to easily implement their own
reward functions or reward modification functions.

Network Module: We have highly abstracted the network structure, enabling our
framework to support various RNN structures such as LSTM (Hochreiter and
Schmidhuber, 1997) and GRU (Cho et al., 2014), transformers (Vaswani et al.,
2017), and even networks from other reinforcement learning frameworks. For
instance, our framework can load models trained using the Stable Baselines3
framework (Raffin et al., 2021).

Algorithm Module: We have abstracted the algorithm component to focus on the
timing of loss calculation and the combination of losses. In Appendix B, we
provide an example of how to add the MAT algorithm (Wen et al., 2022) with
just a few lines of code.

###  2.2 High Performance

In addition to usability, we also place a strong emphasis on the performance
and speed of the OpenRL framework. On a standard laptop, OpenRL can complete
the training of the CartPole task in just a few seconds. Compared to the
RL4LMs framework (Ramamurthy et al., 2022), our training speed for dialogue
tasks has improved by $17\%$, with improvements in various performance
indicators as well (see Appendix C for specific experimental results).

DeepSpeed: OpenRL is one of the few reinforcement learning frameworks that
support DeepSpeed (Rasley et al., 2020), which can help us train larger neural
networks in reinforcement learning tasks and accelerate model parallelism
through multiple GPUs. We present the training results using DeepSpeed in
Appendex D.

Mixed Precision Training & Half-Precision Policy: We support PyTorch’s native
mixed precision training to accelerate the training process. During the data
collection phase, where randomness is inherently required, the demand for
model precision is not as high. Our model supports the use of a half-precision
policy for data collection, thereby accelerating the data collection process.

###  2.3 Usability

Designed with usability in mind, OpenRL supports various flexible usage
scenarios, including dictionary observation spaces and one-click switching
between serial and parallel training environments.

Configuration: OpenRL supports setting hyper-parameters through configuration
files and command-line parameters. We also support the use of global variables
in configuration files, simplifying the process for users to input repeated
parameter values.

Experimental Tracking: We support the use of Weights & Biases (Biewald, 2020)
and TensorBoard (Abadi et al., 2016) for recording experimental data. We have
abstracted this part of the log data, enabling users to record new customized
experimental data without modifying OpenRL’s underlying code.

Gallery: To help users quickly get started with OpenRL and understand its
usage, we provide a Gallery module, which includes complete training codes for
various algorithms and tasks.

Arena: OpenRL provides an Arena module for competitive environments. Agents
trained through OpenRL’s self-play and rule-based agents can compete and be
evaluated in the Arena module.

Support HuggingFace: To fully utilize community resources, OpenRL supports
loading models and datasets from HuggingFace (Hugging Face Inc., 2023). It
also supports loading models trained by the Stable Baselines3 organization
(Raffin et al., 2021) on HuggingFace for testing and training.

Documents: We provide complete bilingual documentation and detailed tutorials.
Our documentation is maintained through the Read the Docs platform, and the
documentation address is: https://openrl-docs.readthedocs.io/.

##  3 Conclusion

In conclusion, OpenRL emerges as a versatile and powerful reinforcement
learning framework that offers modularity, high performance, and user-
friendliness. Its unique features, such as self-play and NLP integration, make
it an indispensable tool for researchers across various levels of expertise.
We are dedicated to its continuous enhancement and fostering a collaborative
community, confident that OpenRL will drive innovation and cross-disciplinary
advances in the dynamic field of reinforcement learning.

Acknowledgments and Disclosure of Funding

We sincerely thank Fanqi Lin, Ruize Zhang, Ye Li, Geo Jolly and many other
open-source contributors for their contributions to the development of OpenRL.
We also genuinely appreciate our users for providing numerous constructive
comments and suggestions.

##  Appendix A OpenRL’s General Code Interface

The following code snippet provides a general interface for our framework:

##  Appendix B Implement New Algorithms: Take MAT algorithm as an Example

The modular design and abstracted algorithm component of OpenRL simplify the
implementation of new reinforcement learning algorithms. Here, we use the MAT
algorithm (Wen et al., 2022) as an example to demonstrate how a new algorithm
can be implemented by adding just a few lines of code to the algorithm module:

##  Appendix C Training Results Compared to RL4LMs

The table below presents the results of training on the dialogue task (Li et
al., 2017) using OpenRL and comparing them with RL4LMs (Ramamurthy et al.,
2022). The results indicate that after training with reinforcement learning,
all model indicators have improved. Additionally, OpenRL demonstrates a faster
training speed compared to RL4LMs (on the same server with NVIDIA 3090 GPUs,
the speed is increased by $17.2\%$), as well as better final performance:

| OpenRL | Supervised Learning | RL4LMs
---|---|---|---
FPS(Speed) | 13.20(+17%) | — | 11.26
Rouge-1 | 0.181(+10%) | 0.164 | 0.169
Rouge-Lsum | 0.153(+12%) | 0.137 | 0.144
Meteor | 0.292(+25%) | 0.234 | 0.198
SacreBLEU | 0.090(+43%) | 0.063 | 0.071
Intent Reward | 0.435(+1.9%) | 0.427 | 0.455
Mean Output Length | 18.69 | 18.95 | 18.83
Table 2: Training results on DailyDialog (Li et al., 2017).

##  Appendix D Training Results of DeepSpeed

We present in the table below some training results using DeepSpeed (Rasley et
al., 2020) within OpenRL. The outcomes demonstrate that employing OpenRL with
DeepSpeed yields faster training speeds compared to using OpenRL with data-
parallel.

| DeepSpeed | Data-Parallel
---|---|---
FPS(Speed) | 5.11(+30%) | 3.94
Number of GPUs | 2 | 2
Memory Usage per GPU(MB) | 13537 | 7207
GPU Type | RTX 3090 | RTX 3090
Batch Size per GPU | 8 | 8
Table 3: DeepSpeed v.s. Data-Parallel for GPT-2-small (Radford et al., 2019). | DeepSpeed | Data-Parallel
---|---|---
FPS(Speed) | 7.09(+35%) | 5.25
Number of GPUs | 4 | 4
Memory Usage per GPU(MB) | 35360 | 15854
GPU Type | NVIDIA A100 | NVIDIA A100
Batch Size per GPU | 8 | 8
Table 4: DeepSpeed v.s. Data-Parallel for OPT-1.3B (Zhang et al., 2022).

## References

  * Abadi et al. (2016) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  _arXiv preprint arXiv:1603.04467_ , 2016.
  * Arulkumaran et al. (2017) Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath.  Deep reinforcement learning: A brief survey.  _IEEE Signal Processing Magazine_ , 34(6):26–38, 2017.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Biewald (2020) Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Burda et al. (2018) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Chen et al. (2022) Wenze Chen, Shiyu Huang, Yuan Chiang, Tim Pearce, Wei-Wei Tu, Ting Chen, and Jun Zhu.  Dgpo: discovering multiple strategies with diversity-guided policy optimization.  _arXiv preprint arXiv:2207.05631_ , 2022.
  * Cheng et al. (2023) Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak.  Extreme parkour with legged robots.  _arXiv preprint arXiv:2309.14341_ , 2023.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Degrave et al. (2022) Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al.  Magnetic control of tokamak plasmas through deep reinforcement learning.  _Nature_ , 602(7897):414–419, 2022.
  * Eysenbach et al. (2018) Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Fu et al. (2020) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hugging Face Inc. (2023) Hugging Face Inc.  Hugging face – the ai community building the future., 2023.  URL https://huggingface.co/.  Accessed: 2023-11-06.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu.  Dailydialog: A manually labelled multi-turn dialogue dataset.  _arXiv preprint arXiv:1710.03957_ , 2017.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Lin et al. (2023) Fanqi Lin, Shiyu Huang, Tim Pearce, Wenze Chen, and Wei-Wei Tu.  Tizero: Mastering multi-agent football with curriculum learning and self-play.  _arXiv preprint arXiv:2302.07515_ , 2023.
  * Margolis and Agrawal (2023) Gabriel B Margolis and Pulkit Agrawal.  Walk these ways: Tuning robot control for generalization with multiplicity of behavior.  In _Conference on Robot Learning_ , pages 22–31. PMLR, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Polydoros and Nalpantidis (2017) Athanasios S Polydoros and Lazaros Nalpantidis.  Survey of model-based reinforcement learning: Applications on robotics.  _Journal of Intelligent & Robotic Systems_, 86(2):153–173, 2017.
  * Qin et al. (2022) Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu.  Neorl: A near real-world benchmark for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35:24753–24765, 2022.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffin et al. (2021) Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 22(268):1–8, 2021.  URL http://jmlr.org/papers/v22/20-1364.html.
  * Ramamurthy et al. (2022) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi.  Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_ , 2022.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506, 2020.
  * Roy et al. (2021) Rajarshi Roy, Jonathan Raiman, Neel Kant, Ilyas Elkin, Robert Kirby, Michael Siu, Stuart Oberman, Saad Godil, and Bryan Catanzaro.  Prefixrl: Optimization of parallel prefix circuits using deep reinforcement learning.  In _2021 58th ACM/IEEE Design Automation Conference (DAC)_ , pages 853–858. IEEE, 2021.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wen et al. (2022) Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang.  Multi-agent reinforcement learning is a sequence modeling problem.  _Advances in Neural Information Processing Systems_ , 35:16509–16521, 2022.
  * Yu et al. (2022) Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu.  The surprising effectiveness of PPO in cooperative multi-agent games.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhang et al. (2020) Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian.  Bebold: Exploration beyond the boundary of explored regions.  _arXiv preprint arXiv:2012.08621_ , 2020.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.


Maximal Information Leakage based Privacy Preserving Data Disclosure Mechanisms
Tianrui Xiao, Student Member, IEEE,
and Ashish Khisti, Member, IEEE
^1 T. Xiao is with Faculty of Electrical & Computer Engineering,
University of Toronto, 10 King's College Road, Toronto, Ontario Canada M5S 3G4
tianrui.xiao at mail.utoronto.ca
^2 A. Khisti is with the Faculty of Electrical & Computer Engineering,
University of Toronto, 10 King's College Road, Toronto, Ontario Canada M5S 3G4
akhisti at ece.utoronto.ca

It is often necessary to disclose training data to the public domain, while protecting privacy of certain sensitive labels.  We use information theoretic measures to develop such privacy preserving data disclosure mechanisms. Our  mechanism involves perturbing the data vectors in a manner that strikes a balance in the privacy-utility trade-off.
We use  maximal information leakage between the output data vector and the confidential label as our privacy metric. We first study the theoretical Bernoulli-Gaussian model and study the privacy-utility trade-off when only the mean of the Gaussian distributions
can be perturbed. We show that the optimal solution is the same as the case when the utility is measured using  probability of error at the adversary. We then consider an application of this framework to a data driven setting and provide an empirical approximation to the Sibson mutual information.  By performing experiments on the MNIST and FERG data-sets, we show that our proposed framework achieves equivalent or better privacy than previous methods based on mutual information.

Privacy preservation, information theoretic privacy, generative adversarial networks, auto-encoders

§ ACKNOWLEDGMENT

The authors would like to thank Vincent Y.F. Tan for his insights and advice in the course of developing this work.

[1]
L. Sweeney, “Simple demographics often identify people uniquely,” 2000.
[Online]. Available:

[2]
A. Narayanan and V. Shmatikov, “Robust de-anonymization of large sparse
datasets,” in Proceedings of the 2008 IEEE Symposium on Security and
Privacy, ser. SP '08.1em plus 0.5em minus 0.4emWashington, DC,
USA: IEEE Computer Society, 2008, pp. 111–125. [Online]. Available:

[3]
G. Smith, “On the foundations of quantitative information flow,” in
Foundations of Software Science and Computational Structures, 12th
International Conference, FOSSACS 2009, Held as Part of the Joint European
Conferences on Theory and Practice of Software, ETAPS 2009, York, UK, March
22-29, 2009. Proceedings, ser. Lecture Notes in Computer Science,
L. de Alfaro, Ed., vol. 5504.1em plus 0.5em minus 0.4emSpringer, 2009, pp. 288–302. [Online]. Available:

[4]
C. Braun, K. Chatzikokolakis, and C. Palamidessi, “Quantitative notions of
leakage for one-try attacks,” Electr. Notes Theor. Comput. Sci., vol.
249, pp. 75–91, 2009. [Online]. Available:

[5]
G. Barthe and B. Köpf, “Information-theoretic bounds for differentially
private mechanisms,” in Proceedings of the 24th IEEE Computer
Security Foundations Symposium, CSF 2011, Cernay-la-Ville, France, 27-29
June, 2011.1em plus 0.5em minus 0.4emIEEE Computer Society,
2011, pp. 191–204. [Online]. Available:

[6]
A. Tripathy, Y. Wang, and P. Ishwar, “Privacy-preserving adversarial
networks,” CoRR, vol. abs/1712.07008, 2017. [Online]. Available:

[7]
J. Hamm, “Minimax filter: Learning to preserve privacy from inference
attacks,” CoRR, vol. abs/1610.03577, 2016. [Online]. Available:

[8]
C. Dwork, “A firm foundation for private data analysis,” Commun. ACM,
vol. 54, no. 1, pp. 86–95, Jan. 2011. [Online]. Available:

[9]
M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and
L. Zhang, “Deep learning with differential privacy,” CoRR, vol.
abs/1607.00133, 2016. [Online]. Available:

[10]
K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private
empirical risk minimization,” J. Mach. Learn. Res., vol. 12, pp.
1069–1109, July 2011. [Online]. Available:

[11]
S. Song, K. Chaudhuri, and A. D. Sarwate, “Stochastic gradient descent with
differentially private updates,” in IEEE Global Conference on Signal
and Information Processing, GlobalSIP 2013, Austin, TX, USA, December 3-5,
2013.1em plus 0.5em minus 0.4emIEEE, 2013, pp. 245–248.
[Online]. Available: <https://doi.org/10.1109/GlobalSIP.2013.6736861>

[12]
J. Hamm, P. Cao, and M. Belkin, “Learning privately from multiparty data,”
CoRR, vol. abs/1602.03552, 2016. [Online]. Available:

[13]
R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in
Proceedings of the 22Nd ACM SIGSAC Conference on Computer and
Communications Security, ser. CCS '15.1em plus 0.5em minus
0.4emNew York, NY, USA: ACM, 2015, pp. 1310–1321. [Online].
Available: <http://doi.acm.org/10.1145/2810103.2813687>

[14]
R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel,
“VIME: variational information maximizing exploration,” in Advances
in Neural Information Processing Systems 29: Annual Conference on Neural
Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain,
D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016,
pp. 1109–1117. [Online]. Available:

[15]
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in
Advances in Neural Information Processing Systems 27: Annual Conference
on Neural Information Processing Systems 2014, December 8-13 2014, Montreal,
Quebec, Canada, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and
K. Q. Weinberger, Eds., 2014, pp. 2672–2680. [Online]. Available:

[16]
T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen,
“Improved techniques for training gans,” CoRR, vol. abs/1606.03498,
2016. [Online]. Available: <http://arxiv.org/abs/1606.03498>

[17]
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel,
“Infogan: Interpretable representation learning by information maximizing
generative adversarial nets,” in Advances in Neural Information
Processing Systems 29: Annual Conference on Neural Information Processing
Systems 2016, December 5-10, 2016, Barcelona, Spain, D. D. Lee, M. Sugiyama,
U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016, pp. 2172–2180.
[Online]. Available:

[18]
J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image
translation using cycle-consistent adversarial networks,” CoRR, vol.
abs/1703.10593, 2017. [Online]. Available:

[19]
A. Makhzani, J. Shlens, N. Jaitly, and I. J. Goodfellow, “Adversarial
autoencoders,” CoRR, vol. abs/1511.05644, 2015. [Online]. Available:

[20]
C. Huang, P. Kairouz, X. Chen, L. Sankar, and R. Rajagopal, “Context-aware
generative adversarial privacy,” CoRR, vol. abs/1710.09549, 2017.
[Online]. Available: <http://arxiv.org/abs/1710.09549>

[21]
S. Asoodeh, M. Diaz, F. Alajaji, and T. Linder, “Information extraction under
privacy constraints,” Information, vol. 7, no. 1, p. 15, 2016.
[Online]. Available: <https://doi.org/10.3390/info7010015>

[22]
S. Asoodeh, F. Alajaji, and T. Linder, “On maximal correlation, mutual
information and data privacy,” in 14th IEEE Canadian Workshop on
Information Theory, CWIT 2015, St. John's, NL, Canada, July 6-9,
2015.1em plus 0.5em minus 0.4emIEEE, 2015, pp. 27–31.
[Online]. Available: <https://doi.org/10.1109/CWIT.2015.7255145>

[23]
I. Issa, S. Kamath, and A. B. Wagner, “An operational measure of information
leakage,” in 2016 Annual Conference on Information Science and
Systems, CISS 2016, Princeton, NJ, USA, March 16-18, 2016.1em plus
0.5em minus 0.4emIEEE, 2016, pp. 234–239. [Online]. Available:

[24]
M. S. Alvim, K. Chatzikokolakis, C. Palamidessi, and G. Smith, “Measuring
information leakage using generalized gain functions,” in Proceedings
of the 2012 IEEE 25th Computer Security Foundations Symposium, ser. CSF
'12.1em plus 0.5em minus 0.4emWashington, DC, USA: IEEE
Computer Society, 2012, pp. 265–279. [Online]. Available:

[25]
M. S. Alvim, K. Chatzikokolakis, A. McIver, C. Morgan, C. Palamidessi, and
G. Smith, “Axioms for information leakage,” in IEEE 29th Computer
Security Foundations Symposium, CSF 2016, Lisbon, Portugal, June 27 - July
1, 2016.1em plus 0.5em minus 0.4emIEEE Computer Society,
2016, pp. 77–92. [Online]. Available:

[26]
I. Csiszár, “Generalized cutoff rates and renyi's information
measures,” IEEE Trans. Information Theory, vol. 41, no. 1, pp.
26–34, 1995. [Online]. Available: <https://doi.org/10.1109/18.370121>

[27]
M. Ben-Bassat and J. Raviv, “Renyi's entropy and the probability of error,”
IEEE Trans. Information Theory, vol. 24, no. 3, pp. 324–331, 1978.
[Online]. Available: <https://doi.org/10.1109/TIT.1978.1055890>

[28]
S. Verdú, “\(\alpha\)-mutual information,” in 2015 Information
Theory and Applications Workshop, ITA 2015, San Diego, CA, USA, February
1-6, 2015.1em plus 0.5em minus 0.4emIEEE, 2015, pp. 1–6.
[Online]. Available: <https://doi.org/10.1109/ITA.2015.7308959>

[29]
I. Issa and A. B. Wagner, “Operational definitions for some common information
leakage metrics,” in 2017 IEEE International Symposium on
Information Theory, ISIT 2017, Aachen, Germany, June 25-30, 2017.1em plus 0.5em minus 0.4emIEEE, 2017, pp. 769–773. [Online].
Available: <https://doi.org/10.1109/ISIT.2017.8006632>

[30]
E. Weisstein, “Normal distribution,” 2002. [Online]. Available:

[31]
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” CoRR,
vol. abs/1312.6114, 2013. [Online]. Available:

[32]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
CoRR, vol. abs/1412.6980, 2014. [Online]. Available:

[33]
D. Aneja, A. Colburn, G. Faigin, L. Shapiro, and B. Mones, “Modeling stylized
character expressions via deep learning,” in Asian Conference on
Computer Vision.1em plus 0.5em minus 0.4emSpringer, 2016, pp.