# Redactable Signature Schemes and Zero-knowledge Proofs: A comparative
examination for applications in Decentralized Digital Identity Systems

Bryan Kumara  The Alan Turing Institute, London, U.K. Email: bkumara, mhooper,
cmaple, thobson, jcrowcroft @turing.ac.uk  Mark Hooper  The Alan Turing
Institute, London, U.K. Email: bkumara, mhooper, cmaple, thobson, jcrowcroft
@turing.ac.uk  Carsten Maple  The Alan Turing Institute, London, U.K. Email:
bkumara, mhooper, cmaple, thobson, jcrowcroft @turing.ac.uk  Timothy Hobson
The Alan Turing Institute, London, U.K. Email: bkumara, mhooper, cmaple,
thobson, jcrowcroft @turing.ac.uk  Jon Crowcroft  The Alan Turing Institute,
London, U.K. Email: bkumara, mhooper, cmaple, thobson, jcrowcroft
@turing.ac.uk

(23 July 2023)

###### Abstract

Redactable Signature Schemes and Zero-Knowledge Proofs are two radically
different approaches to enable privacy. This paper analyses their merits and
drawbacks when applied to decentralized identity system. Redactable
Signatures, though competitively quick and compact, are not as expressive as
zero-knowledge proofs and do not provide the same level of privacy. On the
other hand, zero-knowledge proofs can be much faster but some protocols
require a trusted set-up. We conclude that given the benefits and drawbacks,
redactable signatures are more appropriate at an earlier stage and zero-
knowledge proofs are more appropriate at a later stage for decentralized
identity systems.

##  1 Introduction

A critical aspect of economic growth is the integration of the internet as a
medium for innovation and exchange [16]. It is vital that digital identities
which computationally represent external agents are able to address
stakeholder concerns as each digital identity system comes with its own set of
trade-offs that considers various factors such as security, reliability,
resilience, accessibility, and scalability [17] [16] [5] [2]. A key issue
these systems may consider is the degree to which a single party may exert
control over the operation of the system. Centralized digital identity systems
usually contain a dedicated registration process and store information in a
central database [13]. However, the centralization of authority and storage
carries the risk of abuse such as exclusion and censorship alongside
individual profiling [20].

One approach to address these concerns is to adopt a decentralized identity
system. This approach typically involves the use of transparent, open networks
and protocols with voluntary participation in the attempt to minimize data
collection or formal enrolment [23]. To achieve this, Decentralized
Identifiers (DIDs) and Verifiable Credentials (VC) are typically used. The
former is a globally unique identifier which allows a party to be identified
in a verifiable and persistent manner without the need of a centralized
registry. The latter is an open standard for digital credentials with a
digital signature. A VC can be issued by anyone and can be presented for
verification by everyone. A combination of DIDs and VCs gives individual
parties control of their digital attributes without a central authority [4].

DIDs and VCs rely on some verification mechanism to establish truth. This can
be done through cryptographic protocols such as a digital signature which
ensures a message was created by a known sender without tampering [15]. There
are interesting protocols one can construct using digital signatures, and in
this paper we will discuss Redactable Signature Scheme (RSS). RSS builds on
top of digital signatures by allowing a third party to redact or remove parts
of a message and still derive a valid signature. Another approach is to adopt
Zero-Knowledge Proof (ZKP), which is a mathematical argument that convinces
verifiers that some computation has been performed correctly but reveals
nothing else. ZKPs are complex constructions, and can be implemented with
variants that provide their own set of trade-offs.

Both RSS and ZKPs address the issue of privacy-preserving verification in the
context of decentralized identity systems. Individual identities no longer
need to pass through a centralized system where empowered nodes could abuse
their privilege, but users can also protect their privacy when presenting
their digital identity by retaining agency over what information they would
like to disclose to others. RSS allows one to validly redact parts of a VC,
whilst ZKPs allows users to prove that they meet some access criteria without
having to show their VC at all. These protocols address the issue of privacy
and agency for credential holders. For instance, a decentralized identity
system may be used to check that a user is not below the legal age limit for
purchasing alcohol when entering a bar. The credential holder would present
their VC to a verifier for a check on their age, which is visible in the
attributes section. However, a holder may not want to present the whole of
their credential, which may include other attributes such as address, marital
status, and nationality. RSS allows the redaction of other attributes, leaving
only the birth date on the VC for verification. ZKPs takes it a step further,
with the verifiers accepting a proof that the holder of a VC is older than the
legal age but never seeing the VC itself.

This paper examines the benefits and drawbacks of RSS and ZKPs for VCs. Though
both schemes enable a higher degree of privacy and control for the holder of a
credential, they are very different protocols that impacts the overall system
in various ways. We conclude that given their trade-offs, they can actually be
harmonized by being implemented at different stages of an decentralized
identity system to provide a high level of user privacy to best ameliorate
their shortcomings.

##  2 Background

###  2.1 Redactable Signature Schemes

Public-key cryptography, which encompasses digital signatures and zero-
knowledge proofs, rely on the hardness of solving some mathematical problems.
This paper focuses pairing-based cryptographic techniques, which consists of
three bilinear groups $\mathbf{G}_{1},\mathbf{G}_{2},\mathbf{G}_{T}$ and a
bilinear map $e:\mathbf{G}_{1}\times\mathbf{G}_{2}\rightarrow\mathbf{G}_{T}$.
These groups and mappings satisfies the following properties for some prime
order $p$:

  * •

for all $g\in\mathbf{G}_{1},\tilde{g}\in\mathbf{G}_{2}$ and
$a,b\in\mathbf{Z}_{p}$, $e(g^{a},\tilde{g}^{b})=e(g,\tilde{g})^{a\dot{b}}$;

  * •

for $g\neq 1_{\mathbf{G}_{1}}$ and $\tilde{g}\neq 1_{\mathbf{G}_{2}}$,
$e(g,\tilde{g})\neq 1_{\mathbf{G}_{T}}$;

  * •

the map $e$ is efficiently computable.

There are three different types of pairings - Type 1 pairing where
$\mathbf{G}_{1}=\mathbf{G}_{2}$, Type 2 pairing where
$\mathbf{G}_{1}\neq\mathbf{G}_{2}$ with an efficiently computable homomorphism
$\phi^{\prime}:\mathbf{G}_{2}\rightarrow\mathbf{G}_{1}$, and Type 3 pairing
where $\mathbf{G}_{1}\neq\mathbf{G}_{2}$ with no efficiently computable
homomorphishms $\phi:\mathbf{G}_{1}\rightarrow\mathbf{G}_{2}$ and
$\phi^{\prime}:\mathbf{G}_{2}\rightarrow\mathbf{G}_{1}$. We only consider Type
3 pairing in this paper [6].

The RSS construction we examine relies on following computational assumptions
for its security [6]:

  * •

Symmetric Discrete Logarithm (SDL) Assumption: Given
$(g,g^{a})\in\mathbf{G}_{1}^{2}$ and
$(\tilde{g},\tilde{g}^{a})\in\mathbf{G}_{2}^{2}$, it is hard to recover $a$.

  * •

Extended Decisional Diffie-Hellman (EDDH) Assumption: Given $(\\{g^{a\cdot
c^{i}}\\}_{i=0}^{2n^{2}},\\{g^{b\cdot
c^{i}}\\}_{i=0}^{n-1},\\{g^{c^{i}}\\}_{i=1}^{3n^{2}},\\{g^{d\cdot
c^{i}}\\}_{i=1}^{2n^{2}}\in\mathbf{G}_{1}^{7n^{2}+n+2}$ and
$(\\{\tilde{g}^{c^{i}}\\}_{i=1}^{2n^{2}},\tilde{g}^{d},\\{\tilde{g}^{a\cdot
c^{1}}\\}_{i\in[1,2n^{2}]\setminus[n^{2}-n,n^{2}+n]}$, it is hard to decide if
$z=a\cdot b\cdot c^{n^{2}}+b\cdot d$ or $z$ is random.

With these foundations, we can construct the Pointcheval-Sanders (PS) digital
signature. This signature is re-randomizable, meaning that given a valid
signature one can derive a new version of that signature on the same message
by introducing a random element [26]. It will be a useful property as we
construct RSS from PS. To sign a message
$m=(m_{1},...m_{r})\in\mathbf{Z}_{p}^{r}$, the PS scheme does the following
[6]:

  * •

$Setup(1^{k})$: Given a security parameter $k$, output
$pp\leftarrow(p,\mathbf{G}_{1},\mathbf{G}_{2},\mathbf{G}_{T},e)$ of type 3
bilinear groups.

  * •

$Keygen(pp)$: given the public parameters,
$\tilde{g}\overset{{\scriptscriptstyle\$}}{\leftarrow}\mathbf{G}_{2}$ and
$(x,y_{1},...,y_{r})\overset{{\scriptscriptstyle\$}}{\leftarrow}\mathbf{Z}_{p}^{r+1}$,
compute
$(\tilde{X},\tilde{Y}_{1},...,\tilde{Y}_{r})\leftarrow(\tilde{g}^{x},\tilde{g}^{y_{1}},...,\tilde{g}^{y_{r}})$.
Let the secret key $sk=(x,y_{1},...,y_{r})$ and the public key
$pk=(\tilde{g},\tilde{X},\tilde{Y}_{1},...,\tilde{Y}_{r})$.

  * •

$Sign(sk,m_{1},...,m_{n})$: Generate a random
$h\overset{{\scriptscriptstyle\$}}{\leftarrow}\mathbf{G}_{2}$ and outputs the
signature $\sigma\leftarrow(h,h^{(x+\Sigma y_{j}\cdot m_{j}))}$

  * •

$Verify(pk,(m_{1},...m_{r}),\sigma)$: Given a signature
$\sigma=(\sigma_{1},\sigma_{2})$, check if $\sigma_{1}\neq 1_{\mathbf{G}_{1}}$
and
$e(\sigma_{1},\tilde{X}\cdot\Pi\tilde{Y}_{j}^{m_{j}})=e(\sigma_{2},\tilde{g})$
are satisfied. If it is, return $1$ else return $0$

For proofs of correctness and security alongside a more comprehensive overview
of the PS signature scheme, readers should refer to [6]. Given the PS
signature scheme, one only needs slight modification to reach the RSS
construction. The idea of this RSS is that given a set of messages
$\\{m_{i}\\}_{i\in I}$ and an associated signature $\sigma$, one can redact
parts of a message $i\in\overline{I}$ for some $I\subset[1,n]$ and
$\overline{I}=[1,n]\setminus I$ such that the signature is valid for
$\\{m_{i}\\}_{i\in I}$. For a more comprehensive treatment of RSS that covers
alternative constructions and features, readers should refer to [1].

Intuitively, this RSS construction uses the PS signature scheme and aggregates
the redacted messages into one element. To prove that this element was
generated honestly a second element is calculated from the first. To avoid a
quadratic scaling based on the length of the message that previous
constructions RSS suffered from, this scheme starts from a modified instance
of the PS scheme where the secret key contains two scalars $x,y$ and a hash
function is used to create scalars that indicate a commitment on which indexes
$i\in[1,n]$ have been redacted, preventing a maliciously generated signature.
A second pairing equation is added for the verification process to check the
scalars generated from the hash function. This results in a constant signature
size and a public key that scales linearly to the message length $n$, which is
a notable improvement from previous RSS schemes. A full examination of this
RSS construction covering security proofs can be found in [26].

Given a message of length $n$, the RSS Scheme we consider proceeds as follows:

  * •

$Keygen(1^{k},n)$: Given a security parameter $1^{k}$ and an integer $n$,
generate two random elements
$(g,\tilde{g})\leftarrow\mathbf{G}_{1}\times\mathbf{G}_{2}$ and scalars
$(x,y)\leftarrow\mathbf{Z}_{p}^{2}$. Then, compute the following:

    1. 1.

$\tilde{X}\leftarrow\tilde{g}^{x}$

    2. 2.

$\tilde{Y_{i}}\leftarrow\tilde{g}^{y_{i}}\forall 1\leq i\leq n$

    3. 3.

$Y_{i}\leftarrow g^{y_{i}}\forall i\in[1,n]\cup[n+2,2n]$

The secret key $sk$ is $(x,y)$ and the public key $pk$ is
$(H,g,\tilde{g},\\{Y_{i}\\}_{i=1}^{n},\\{Y_{i}\\}_{i=n+2}^{2}n,\tilde{X},\\{\tilde{Y}_{i}\\}_{i=1}^{n})$
where $H:\\{0,1\\}*\rightarrow\mathbf{Z}_{p}^{*}$ is the description of the
hash function.

  * •

$Sign(sk,\\{m_{i}\\}_{i=1}^{n})$: Given a secret key $sk$ and $n$ messages
$m_{1},...,m_{n}$, generate a random element
$\sigma_{1}\overset{{\scriptscriptstyle\$}}{\leftarrow}\mathbf{G}_{1}$ and
compute $\sigma_{2}=\sigma_{1}^{x+\Sigma_{i=1}^{n}y_{i}\cdot m_{i}}$. Let
$\sigma_{3}=1_{\mathbf{G}_{1}}$ and $\tilde{\sigma}=1_{\mathbf{G}_{2}}$.
Output a signature $\sigma=(\sigma_{1},\sigma_{2},\sigma_{3},\tilde{\sigma})$.

  * •

$Derive(pk,\sigma,\\{m_{i}\\}_{i=1}^{n},\mathcal{I})$: Given an index
$\mathcal{I}\subseteq[1,n]$ of message segments we would like to retain, a
message $m$, its associated signature $\sigma$, and a public key $pk$,
generate two random scalars $(r,t)\leftarrow\mathbf{Z}_{p}^{2}$ and compute
the following:

    * –

$\sigma_{1}^{\prime}\leftarrow\sigma_{1}^{r}$

    * –

$\sigma_{2}^{\prime}\leftarrow\sigma_{2}^{r}\cdot(\sigma_{1}^{\prime})^{t}$

    * –

$\tilde{\sigma}^{\prime}\leftarrow\tilde{g}^{t}\cdot\Pi_{j\in\overline{\mathcal{I}}}\tilde{Y}_{j}^{m_{j}}$

where $\bar{\mathcal{I}}=[1,n]\setminus\mathcal{I}$. For all
$i\in\mathcal{I}$, compute the following scalar $c_{i}\leftarrow
H(\sigma_{1}^{\prime}||\sigma_{2}^{\prime}||\tilde{\sigma}^{\prime}||I||i)$
which is used to calculate:

    * –

$\sigma_{3}^{\prime}\leftarrow\Pi_{i\in\mathcal{I}}[Y_{n+1-i}^{t}\cdot\Pi_{j\in\overline{\mathcal{I}}}Y_{n+1-i+j}^{m_{j}}]^{c_{i}}$

The derived signature is
$\sigma^{\prime}=(\sigma_{1}^{\prime},\sigma_{2}^{\prime},\sigma_{3}^{\prime},\tilde{\sigma}^{\prime})$
for a redacted message $\\{m_{i}\\}_{i\in\mathcal{I}}$.

  * •

$Verify(pk,\sigma,\\{m_{i}\\}_{i\in\mathcal{I}})$: Given a signature $\sigma$,
a public key $pk$, and a message $\\{m_{i}\\}_{i\in\mathcal{I}}$, if
$\sigma_{1}=1_{\mathbf{G}_{1}}$, return $0$. Otherwise, test if both equations
hold before returning $1$:

    * –

$e(\sigma_{1},\tilde{X}\cdot\tilde{\sigma}\cdot\Pi_{i\in\mathcal{I}}\tilde{Y}_{i}{m_{i}})=e(\sigma_{2},\tilde{g})$;

    * –

$e(\sigma_{3},\tilde{g})=e(\Pi_{i\in\mathcal{I}}Y_{n+1-1}^{c_{i}},\tilde{\sigma})$;

Note that the public key of the issuer is sufficient to generate a derived
signature, which allows a holder of a credential redact and derive signatures
without informing the issuer which is just the feature we’d like users to have
in a decentralized ecosystem.

###  2.2 Zero-Knowlege Proofs

Zero-knowledge proofs (ZKPs) are methods where a party called the prover can
prove to another party called the verifier that a given statement is true
without revealing any additional information [22]. Traditionally, a prover can
show knowledge of a certain piece of information by simply revealing it.
However, ZKPs allows one to prove possession of certain secret knowledge
called a witness which satisfies some statement without revealing the witness
itself or any additional information [22]. ZKPs have a diverse range of
constructions: some require an initial set-up phase, represent computations
differently, and some protocols may involve multiple rounds of interaction
between the prover and the verifier [22].

For the purposes of this paper, we consider a type of ZKP called Zero-
Knowledge Succinct Non-Interactive Argument of Knowledge (ZK-SNARK) which are
efficient in terms of size and verification time (Succinct), does not require
multiple rounds of communication between the prover and the verifier (Non-
interactive), and provides sound proofs against a polynomial-bounded prover
(Arguments of Knowledge) [24]. Theoretically, a protocol is defined as zero-
knowledge with regards to a simulator. This is a machine that operates in a
theoretical world that is indistinguishable from the real world for a party
that attempts to decide if knowledge was leaked. In this theoretical world,
the the simulator can compute a proof without anything else but by just
knowing the truth of the statement. A protocol is then zero-knowledge if a
proof produced by a simulator who does not know the witness is the same as one
produced by a real-world prover that knows the witness [24].

The efficiency of ZK-SNARKs in terms of size, speed, and communication makes
this an attractive protocol for anonymous disclosure. However, ZK-SNARKs
require an initial set-up phase that requires trust. This procedure generates
a common set of parameters called the common reference string (CRS) that must
be used every time we run a ZK-SNARK proof. CRS generation requires trust as
parties involved must delete by-products of the procedure called toxic waste
which may otherwise be used to generate forged proofs [24]. Formally, a ZK-
SNARK uses some efficiently decidable binary relation $\mathcal{R}$ with pairs
$(u,w)\in\mathcal{R}$ where $u$ is called the statement and $w$ is the
witness. A non-interactive argument for $\mathcal{R}$ consists of
probabilistic polynomial algorithms $\Pi=(G,P,Ver,Sim)$ where:

  * •

$(crs,vrs,td)\leftarrow G(1^{\lambda},\mathcal{R})$: Given a securty parameter
$\lambda$ and a relation $\mathcal{R}$, output a common reference string
$crs$, a trapdoor $td$, and a verification key $vrs$.

  * •

$\pi\leftarrow P(crs,u,w)$: Given a common reference string $crs$, a statement
$u$, and a witness $w$, the prover algorithm outputs some argument $\pi$.

  * •

$b\leftarrow Vers(crs,u,\pi)$: Given an input statement $u$, and argument
$\pi$, and a common reference string $crs$. the verifier algorithm outputs
$b=1$ if the proof is accepted and $b=0$ otherwise.

  * •

$\pi\leftarrow Sim(crs,td,u)$: Given a trapdoor $td$, a statement $u$, and a
proof $\pi$, the simulator returns an arguement $\pi$.

.

An in-depth exploration into the exact procedures of ZK-SNARKS is complicated
and lies outside the scope of this paper. [29] [21] [18] [14] [24] are
particularly good at providing an accessible introduction into this protocol.
In short, zk-SNARKs operate by representing computation as a sequence of
constrained variables. This is then transformed to polynomials. With the
addition of some hiding techniques and using the witness to create a modified
set of polynomials, we can map them to pairing-friendly groups. A verifier
simply needs to evaluate some pairing functions to check a proof. Informally,
ZK-SNARK is able to represent any computation that can be verified quickly,
and this allows us to express complex access criteria [24]. There are multiple
ways to implement ZK-SNARKs, and this paper considers the Groth16
implementation. This is a general-purpose ZK-SNARK which enables constant size
proof and fast verifier time but requires a circuit-specific trusted setup for
the pre-processing phase [9]. For circuit of arithmetic operations, a Groth16
proof consists of 2 elements in $\mathbf{G}_{1}$ and 1 element in
$\mathbf{G}_{2}$ which is an improvement from previous implementation such as
the Pinocchio ZK-SNARK which contains 7 elements in $\mathbf{G}_{1}$ and 1
element in $\mathbf{G}_{2}$ [9]. Verification in Groth16 only requires 3
pairings instead of the 12 used in Pinocchio and comes with a shorter CRS [9].
In addition to the appealing performance metrics, Groth16 also allows linkage
proofs [19]. Linkage proofs show that a private set of Groth16 proofs share
some subset of hidden public inputs. Verifying a linkage proof does not reveal
the Groth16 proof constituents or even the public inputs involved. Linkage
proofs are key in constructing the ZKP system we are interested in [19].

zk-Creds is a ”flexible, issuer-agnostic anonymous credential toolkit for
complex identity statements” [19]. The system relies on Groth16 ZK-SNARKs to
prove these identity statements, and linkage proofs of Groth16 allows zk-Creds
to work with different forms of credentials as long as they have some public
credential. This enables post-deployment composition of credentials, and new
access criteria. The assumption made by zk-Creds is that there is a list of
issued credentials that is either maintained by a trusted party or some
decentralized system [19], which is the standard assumption made by VCs.

zk-Creds allows three sets of functionalities: issue, show and verify, and
revoke [19]. Unlike RSS and VCs in general, zk-Creds credentials does not rely
on signatures but rather proof of being in an issued set. In the issuance
process, a user requests the issuer that a credential should be issued by
meeting some predetermined requirement. Once the credential is placed on the
list of issued credential, the user can use it to prove that some access
criteria is met to other parties. Using a credential provides a membership
proof of the credential being on the list of issued credentials, and a proof
that committed attributes of credentials meets the access criteria. A
credential can be revoked simply by removing it from the list of issued
credentials. In [19] notes that there are multiple ways to demonstrate set
membership, and uses Merkle forests as a new approach to gain a large
reduction in proof generation time but a slight increase in verification time
and witness size. An examination of alternative approaches can be found in
[12].

Groth16 linkage proofs can be combined with membership proof (which shows that
an element belongs to a set) and multiple access proofs (which shows that some
criteria is fulfilled by a prover) to create Blind Groth16 [19]. Blind Groth16
allows us to reuse membership proofs in multiple showings and still maintain
privacy [19]. This also allows an easy composition of access criteria to
prevent determining in advance all the circuits that verifiers will support,
prevent the generation of circuit parameters for every combination of access
criteria, and avoid dynamical generation of circuit parameters for access
criteria as they are needed. Consequently, Blind Groth16s allows one to show
that various Groth16 proofs are made based on the same credential without
revealing the credential. A full exploration of zk-Creds can be found in [19].

##  3 Concrete Performance Metrics

Previous work on RSS have generally examined its performance with respect to
the number of arithmetic operations required, or from a theoretical
computational complexity. However, to better assess their suitability for VCs,
we would need concrete metrics such as run-time and size in terms of
millisecond and bytes respectively. Thus, we have gathered results on both RSS
and zk-Creds for VCs when implemented using the Rust programming language.

The VC we considered is compliant with W3C standards which can be found in
[7]. This sample VC details a user’s personal information such as their name,
date of birth, and address (which consists of street name, postcode, and
country). For RSS, we want to create a redacted VC which shows that a user
resides in a certain country and nothing else. Thus, all the fields but the
name the country have been redacted which should give us a good illustration
of near-worst-case performance as users redact leave only two fields
unredacted. The code for our implementation can be found on [25]. As this is
merely a proof-of-concept, the implementation would be unsuited for immediate
deployment in a VC system, and only unit testing has been conducted to ensure
that components run as intended.

We collect the speed and size of the RSS algorithm discussed in the previous
section. We have 4 algorithms to measure (generating keys, initial signing,
deriving a redaction signature, and verifying signatures), gathering the size
of the signature, keys, and verification elements, and the results are
presented below, rounded to 2 significant figures:

Category | Runtime (ms) | Size (bytes)
---|---|---
public key | 588.22 | 816
private key | 7.01 | 112
signature | 545 | 1344
derived signature | 824.84 | 1344
verifying signature | 310.77 | 32

For zk-Creds, we use the performance results gathered by [19]. Their
implementation focused on converting an existing credential into zero
knowledge proofs and demonstrating a range of access criteria such as expiry,
late limiting, and clone resistance. The results below gives prover-side
optimized worst-case cost where the cost of showing a credential is in a
membership list is included:

Client-Opt | Runtime (ms) | Proof Size (bytes)
---|---|---
Simple Possession | 784 | 744
Expiry | 875 | 1064
Linkable Show | 879 | 1064
Rate Limiting | 895 | 1064
Clone Resistance | 916 | 1064
VerifyShow | 6 | 1064

The following results are after excluding the cost of the membership list but
still prover-side optimized:

Client-Opt | Runtime (ms) | Proof Size (bytes)
---|---|---
Simple Possession | 5 | 744
Expiry | 98 | 1064
Linkable Show | 104 | 1064
Rate Limiting | 117 | 1064
Clone Resistance | 139 | 1064
VerifyShow | 6 | 1064

Lastly, [19] has results where computations are optimized for the verifier.
This covers the scenario where a SNARK proof is not reused, and a batch
verification is included which indicates the throughout of verifying a set of
100 proofs. All verification numbers are single-threaded and concurrent
processing is possible:

Server-Opt | Runtime (ms) | Proof Size (bytes)
---|---|---
Simple Possession | 699 | 744
Expiry | 796 | 1064
Linkable Show | 837 | 1064
Rate Limiting | 817 | 1064
Clone Resistance | 812 | 1064
VerifyShow | 1.5 | 192
Verify Batch | 1.8 verifs | 192

##  4 Evaluation

Having gathered concrete metrics from RSS and zk-Creds, we can discuss the two
schemes in greater depth for VC applications.

###  4.1 RSS for VC

The RSS construction we considered possesses several attractive properties in
theory. Namely, a constant signature size, flexible attribute redaction for
selective disclosure, no requirements for a trusted set-up, and possible
schemes built on top of it. However, it has some key drawbacks. From the
results gathered, the constant signature and private key size of 1344 bytes
and 112 bytes respectively are substantially bigger than other signature
schemes used in VCs. For instance, the pairing-based BBS+ signature scheme
that is in the final stages of standardization has a signature size of 112
bytes and a private key size of 32 bytes alongside a 96 byte public key when
instantiated on the widely used BLS12-381 parameters [28] [3]. This is a far
more attractive in terms of size, and BBS+ signatures provide proof-of-
knowledge schemes that also enables selective disclosure of messages similar
to ZK-SNARKs. Given our implementation scenario that considered redacting all
but one’s name and country on a VC, the BBS+ proof would have a size of 464
bytes. Again, in terms of size, this is a significantly more attractive result
than RSS by offering half the size of a derived signature.

Regarding the run-time, most of the RSS algorithms are much slower than BBS+.
For instance, generating keys takes 151.81ms for BBS+ but 588.22ms for RSS
public keys. RSS key generation has the potential to run much slower too,
given that public keys scale linearly with the size of a message. In addition,
signing using the BBS+ scheme takes only 57.02ms which is almost 10x faster
than the RSS signature and almost 15x faster than generating a derived
signature. The only superior performance for RSS is in the verification, where
RSS takes 310.77ms to verify whilst verifying a BBS+ proof takes 3478.74ms.
This is a roughly 10x difference, but in absolute terms the slowness of BBS+
proof verification will be much more noticeable to users than the comparative
difference in BBS+ and RSS signing. The issue of the BBS+ proof verification
is also more significant when we consider that the BBS+ proof verification has
a linear scaling. Considering a VC with significantly more attributes than our
implementation, going beyond 3.5 seconds to verify may become a barrier for
adoption.

Though the public key of RSS will scale with the length of the message, it is
arguable that given the rare frequency of generating keys this is an
acceptable one-off cost unlike the linear growth of a BBS+ proof which has to
be used every single time one wants to perform selective disclosure. Thus, if
speed is a concern the verification time of RSS may make it preferable than
adopting BBS+ signatures.

However, there are other considerations outside of the concrete metrics from
RSS. On a general level, RSS offers credential holders a level of flexibility
with what information to disclose. Returning to the canonical example of
showing that one is above 18 years old, an individual may not want to show any
other fields of a VC except for their age. However, other users may have less
concerns and be comfortable with displaying their whole or partial VC. RSS
allows users to chose their own level of privacy, and this will impact the
run-time to generate a derived signature. RSS also does not require a trusted
set-up as used in ZK-SNARK proofs. Of course, trust is placed in the issuer of
a credential and is a form of centralization, but this form of trust is also
placed in a ZK-SNARK anyway where one has to assume that a digital credential
is generated by a trustworthy issuer. The lack of a trusted set-up for an RSS
scheme means that there is no risk of a forged proof (however small) due to
kept toxic waste of a CRS, and a forgery attempt of a redacted signature is
thwarted by the security properties of an RSS construction as seen in [26].

Lastly, RSS can also be used as a building block for more advanced
cryptographic schemes beneficial to VCs. [26] used the RSS implementation
considered on this paper to create a group signature (which permits members of
a group of anonymously sign a message which represents the group) with a
revocation mechanism that enables users to be removed from a group. One of the
uses of this feature is for public transport, specifically to implement an
anonymous transport subscription pass where users are able to take an
unlimited number of trips within a fixed period. Given that VC applications
may include that of government registries [23], an anonymous travel pass
within a subscription model for public transport would be a perfect fit for VC
functionality.

However, there are some more drawbacks for RSS. In addition to the slower and
bigger key generation and signature metrics compared to BBS+, validating an
RSS requires receivers to perform their own computation on top of the RSS
scheme. Though RSS lends itself to convenient constructions as highlighted
above in the case of an anonymous transport subscription, it is not as
flexible as zk-Creds as we will see in the following sections. Though the
public key linear growth is not ideal, it is an improvement over previous
quadratic key sizes of the RSS [1] and again keys do not have to be frequently
generated which may be more efficient than the linear growth of BBS+ proof
verification scheme. With execution times running at 800ms in the worst case,
this is still within the realm of use-ability for DID and VC application. The
format of a W3C VC is relatively small with around 6 or so attributes, and
unless substantially more attributes would be included it would not
drastically impact performance or user experience.

###  4.2 zk-Creds for VC

On the other hand, zk-Creds possess desirable properties for VCs unavailable
to RSS. The Groth16 ZK-SNARK has one of the smallest proof size and fastest
verification time in ZKP literature with a competitive proof generation [19].
Indeed, based on the results from [19], zk-Creds runs with a similar speed
when we work with generating a proof alongside its membership witness compared
to RSS. For instance, access criteria shown above range from 784ms to 916ms
when working with a client-side optimization. In fact, it has a slightly
better performance when we work with a server-optimized configuration (though
a difference of 100ms would hardly be noticeable in real-life applications).
Verifying proofs takes anywhere from 1.8ms to 6ms, is a huge performance
improvement over RSS and BBS+. With proof sizes ranging anywhere from 192
bytes to 1064 byes, its clear that the speed and size of zk-Creds is superior
when compared to RSS and BBS+. Bear in mind that these results already pass
the access criteria check, whereas in RSS a verifier would still need to
extract the information from the VC and perform the access check after
verifying the redacted signature. The use of sub-circuits called gadgets also
ensures that verifiers can create separate proofs that allow pre-computation
to further improve speed. Gadgets also allows the representation of more
complex access criteria such as session-binding and rate limiting which
provides a more fine-grained access control than what is possible with RSS, so
there is room for optimization to derive even better performance metrics over
RSS .

In addition to performance, Groth16 linkage proofs is an extremely valuable
mechanism which allows proofs to show that a hidden collection of Groth16
proofs share hidden public inputs. Given the VC assumption of having a
publicly verifiable identity, one could chain different credentials (for
instance, a digital driving license, digital work visa, digital marriage
certificate) the same way as physical identity documents for an individual all
refer to the same person. Possible extension of linkage proofs can be applied
internationally between countries to acknowledge travel documents issued by
another authority. zk-Creds has shown that it is possible to utilize existing
government-issued credentials such as an NFC-enabled passport into a digital
credential which is hugely beneficial in easing adoption. This would not be
possible with RSS, as different credentials would have to be redacted
separately and sent for verification or verifiers must be re-configured to
accept different verification methods. However, the main barrier of zk-Creds
multiple-credential linkage is the trust assigned by authorities to
credentials issued by other parties. This is a largely political and social
issue that lies outside the scope of this paper, but is worth mentioning in
the context of inter-systems interactions.

The use of Groth16, in addition to the linkage proofs that enables
interoperability, also provides other desirable properties. Like the RSS
construction considered, Groth16 allows re-randomization with a weak form of
simulation extractability. This guarantees that even if a malicious party has
seen some proofs before, it cannot prove a new statement without knowing the
witness. Given how frequent proofs may be required for decentralized
identities, some form of information leak must be anticipated. With Groth16,
should an adversary see old proofs they would still be unable to create forged
proofs without the witness. This is a desirable feature to prevent the
’capture’ of information which is possible in RSS as their verification still
involves information extraction from a given VC.

Given how crucial digital identities are, it is not surprising that the
biggest drawback of zk-Creds is the trusted set-up procedure, followed by the
practical integration of different credentials to take advantage of the
linkage proofs enabled by Groth16. As mentioned, every application using
Groth16 must have an initial trusted set-up phase which would need to be
conducted after issuing a VC for every user. In addition to keeping the toxic
waste of a CRS, any small errors in the whole protocol as demonstrated by an
incorrect input aliasing can lead to falsifiable proofs which would be fatal
to the implementation of a trust-worthy digital identity system. For instance,
security audit company Beosin discovered that the zk-SNARK project Semaphore
where the uin256 type represented a larger range of values than what is
accepted in the arithmetic circuit, leading to proofs that can be used more
than once and therefore enabled double spending [Beosin]. There have been some
notable improvements to the toxic waste to address these concerns. [27]
developed a distributed trusted set-up that is secure as long as at least one
of the parties have honestly deleted their toxic waste. Though [27] notes that
there is an issue of robustness as all parties must be fixed before starting
the CRS and must be active throughout the whole trusted set-up phase, this may
not be a significant barrier for a VC depending on the implementation and
connectivity. Further improvements such as [31] was able to avoid using the
random beacon model to further simplify the process. Note that not all of the
trusted set-up improvements are ready for deployment with some being in the
theory phase so a more nuanced discussion will be needed. The issue of fatal
errors in the set-up phase is a more practical one, and the best mitigation
would be an open-source approach with consistent audits. However, it is
possible to remove the trusted set-up entirely by replacing the ZK-SNARK
component of zk-Creds with another protocol such as ZK-STARK which does away
with this phase. The trade-off would be a significantly slower performance
which may inhibit the system.

The last drawback is that of witness generation. As mentioned in previous
sections, zk-Creds uses witnesses to generate proofs by the prover. Witnesses
in zk-Creds are specifically based on the credential issued list. However, the
witness must always be generated from the most recent list of issued
credential to ensure that a credential is still valid at the time of
generating a proof. Though this does not seem to be problematic in theory, in
practice it may make operations such as pre-processing difficult if there is a
high volume of credentials being issued at a time from multiple issuers which
requires the issuer list to be frequently updated. Users with multiple
credentials would have to keep a witness per issuer on top of this. Though
[19] proposes some form of batching addition to potentially mitigate this
issue, it still makes the capability to pre-compute proofs cumbersome. In
addition, there are also privacy considerations for low-update and low-use
systems. Generating a new witness may be matched to a credential as generating
a witness requires showing a credential. This may be matched to a given proof
if there is only a low number of witness generation and participants. Again,
batching approach may mitigate this, but would be a context-specific
deployment. Realistically, given the context of decentralized ID systems, this
issue may be less applicable due to the target volume of users and frequency
of anonymous disclosure. That being said, without the ability to pre-compute
witnesses, it may not be the worst aspect as performance indicates zk-Cred
proof generation will run similarly to RSS.

###  4.3 Verdict

Having considered the merits and drawbacks of each approach, we can conclude
its suitability for decentralized identities. Clearly, there are aspects where
zk-Creds provide advantage that RSS cannot replicate, and vice versa. In the
case of zk-Creds, it is able to provide a much higher level of privacy as
verifiers will never have to see the credential itself, allow more complex
access criteria, and a superior performance compared to RSS. However, the
issues of a trusted-setup and witness generation are factors that deter
adoption. On the other hand, RSS has runs on a competitive speed and size when
compared holistically against similar signatures with a customizable level of
user privacy without the aforementioned issues facing zk-Creds. Yet, it cannot
match zk-Creds in terms of the level privacy available and leaves the burden
of information extraction for access criteria checks to the verifier.

Having gathered concrete performance metrics in addition to considering the
cryptographic features, we conclude that RSS is a much more convenient
implementation for decentralized ID systems that should be adopted first. This
does not mean that zk-Creds should be ruled out entirely. There are good use
cases for them, but the current drawbacks indicate that this protocol is best
adopted at a later stage. RSS is a more attractive scheme to adopt at an
earlier stage of a decentralized ID system in part due to its ease to
implement but also being able to offer an acceptable level of privacy.
Allowing users the ability to selectively redact credential attributes aligns
with the self-sovereign concept of decentralized identity by empowering users
to select the level of privacy that they are comfortable with. This is still a
partial solution for privacy, which is why in the following section we discuss
the benefits of zk-Creds. But as a starting point, RSS performance metrics
already compares well against signature schemes used in the decentralized
identity space so we would not be looking at a radically different performance
metrics. The RSS drawback of attribute verification needing to be computed by
a verifier outside of the signature is also not necessarily an issue. At an
earlier stage of development, it is likely that simpler access criteria will
be more applicable such as proving one’s age, one’s country, etc. The
computation cost imposed on such checks (range checks, string matching) is not
costly and would be expected to add only a negligible overhead on the whole
procedure. The RSS protocol can also be used to construct a mildly complex
limited subscription credential as seen in [26], so more advanced uses of
digital credentials is entirely possible with only RSS.

However, the benefits of RSS can only extend so far. At a later or more mature
stage of development, zk-Creds would be the more appropriate system. zk-Creds
offers much better privacy by using proofs to avoid verifiers from seeing the
actual credential itself. There is no need for credential holders to select
which part of a VC they would like to redact, as the verifiers will never see
any part of it in the first place but be provided with a verifiable proof that
their access criteria is met. This is a much more complicated protocols
(especially through the use of ZK-SNARKS which heavily involves pairing-based
cryptography, arithmetic circuits, and homomorphic mappings) that will require
much more awareness between stakeholders instead of the simpler concept RSS
provides before adoption. Avoiding the implementation issues that trusted set-
ups are liable to stringent checks will also take time, and a feedback process
must be more intensive due to the possible cost of falsifiable proofs whereas
RSS security rests on simpler digital signatures.

As discussed in the previous section, the risk-prone trusted set-up mechanism
can be made more secure through a distributed variant [27]. Though some
balance needs to be struck with regards to the number of participants in the
interest of performance, a system that starts out with zk-Creds as the main
verification protocol but few initial participants is at a higher risk of
toxic waste not being deleted by any parties than one with much more
participants involved at the trusted set-up phase happening at a later stage.
A later stage adoption of zk-Creds would also be friendlier for witness
generation. Given the expected increase in the number of participants,
witness-batching is more feasible to preserve user anonymity, which is a
possible privacy leak as mentioned in the previous section when there is not
enough participants to obscure the witness request and proof verification
processes. One may also be less liable to constant updates on the issued list
of credentials depending on usage. For instance, a digital voter ID may see a
smaller stream of users registering at a later stage instead of a persistent
addition at the first stages of using this system. Some form of registration
batching would also be possible depending on the exact numbers, which would
bring down the number of updates to a credential list making witness updates
less frequent.

Adopting zk-Creds at a later stage is possible due to the Groth16 linkage
proofs. Without this feature, one would have to start from scratch again when
adopting a new protocol for verification which is a notable push-factor for
improving identity systems. Linkage proofs, as demonstrated in [19] using US
NFC-enabled passports, allows parties to build on top of existing identity
infrastructures without changing how they operate. It is therefore possible to
initially adopt RSS as the basis for a decentralized ID verification but later
transition to zk-Creds as parties look into working with different credentials
and attributes that were not initially a part of the system with no material
cost to modifying the system. This is not a feature that is possible with RSS,
where verifiers would have to change their verification protocol to accept
other verification methods used by different credentials. Crucially, linkage
proofs also preserve privacy of other credentials and attributes, and would go
a long way in enabling interoperability as other issuers would mitigate
privacy concerns as public inputs can be hidden when forming proofs.

In short, given the performance metrics and cryptographic features of both
protocols, we conclude that RSS is more appropriate to utilize during the
earlier stages of an identity system whilst zk-Creds is better suited for a
later-stage deployment which can be implemented without any changes to the
initial system but rather to build on top of it. Additionally, a later-stage
deployment of zk-Creds mitigates some issues regarding trusted set-ups and
witness generation due to the number of participants and batching techniques
available respectively whilst RSS provides user-enabled privacy at a
competitive performance metric that can be implemented much earlier in
decentralized identity systems without exposure to risks that zk-Creds are
vulnerable to at an earlier stage.

##  5 Conclusion

This work has examined on two privacy-preserving protocols for verification in
decentralized identity systems. RSS allows users to redact attributes of a
credential, which enables users to select which attributes they would like to
present to a verifier. zk-Creds is a protocol that uses ZK-SNARKS to present
verifiers with proofs that shows an access criteria is met but without having
to disclose their credential. Though both utilize pairing-based cryptography,
they offer different benefits and drawbacks for decentralized identity
systems.

Having implemented these two protocols on a sample VC, we find that RSS offers
competitive performance in terms of speed and size when compared against other
signature schemes currently being standardized for VCs. In addition, RSS
empowers users by allowing them the flexibility to choose what level of
privacy they would be comfortable with and can be used to build anonymous
subscription-based access with applications in public transport. However, RSS
still moves the burden of verifying access criteria to the verifier and would
have difficulties when considering interoperability. We argue that these
considerations makes RSS a good candidate for an early-stage ID system. It is
able to offer user privacy, but the expected simpler access criteria would not
impinge performance and interoperability considerations may done at a later
stage.

Here is where zk-Creds come into play. Through ZK-SNARKs and linkage proofs,
it is able to operate with different credentials and does not compromise on
privacy. Verifiers will never see the actual credential of a user but can set
complex access criteria. In addition, zk-Creds does not need any signing
mechanism as the verification relies on a witness based on a path on a Merkle
tree. There are concerns about the trusted set-up requirement of zk-Creds and
witness generation which may accidentally leak a user’s identity. These
factors make zk-Creds better suited for a later-stage adoption where
interoperability, maximum privacy, and complex access-criteria can be handled
by the protocol. The issues of trusted set-ups and witness generation are also
partially mitigated with a later-stage implementation as a distributed variant
with more participants lessens the risk of toxic waste being kept and less
frequent sign-ups can lead to batching to remove the need to constantly update
the witness. Thus, both techniques need not exist in opposition, but would be
synergistic when implemented at different stages of a decentralized identity
system.

Though our conclusion presents a cohesive application for these two
techniques, there are several areas of RSS and ZKPs that merit future
investigation. One direction of exploration is to examine any post-quantum
variants for the two techniques. The US National Institute of Standards and
Technology (NIST) has recently wrapped up its first competition of quantum-
resistant cryptography. In this competition, lattice-based and hash-based
protocols have been selected but future rounds are being conducted to expand
the range of mathematical structures utilized by considering code-based
cryptography.

Currently, we know of a lattice-based and a hash-based variant of the RSS [11]
[10]. Even within these standards, there are variants of the structure that
can provide differences. For instance, Learning with Errors is a popular
hardness problem used to create lattice-based protocols. But, there are also
variants that constructs Learning with Error over a Ring and a Module [8].
There are also other structures that may be quantum-secure such as isogenies.
Within ZK-SNARKS, there is a lattice-based variant [30]. Again, it is worth
investigating alternative constructions to ensure preparedness against
potential quantum-adversaries that may break the current security assumptions
of RSS and ZK-SNARKs.

##  6 Acknowledgments

This work was supported, in whole or in part, by the Bill and Melinda Gates
Foundation [INV-001309]. Under the grant conditions of the Foundation, a
Creative Commons Attribution 4.0 Generic License has already been assigned to
the Author Accepted Manuscript version that might arise from this submission.

## References

  * [1] Kai Samelin Arne Bilzhause  “Position Paper: The Past, Present, and Future of Sanitizable and Redactable Signatures”  In _International Conference on Availability, Reliability and Security_ Association for Computing Machinery, 2017
  * [2] Joseph J. Atick  “Digital identity: The Essential Guide”  ID4Africa, 2013
  * [3] “BBS+ Implementation” Accessed: 2023-03-30, https://github.com/mattrglobal/bbs-signatures
  * [4] Zoltan Andras Lux Dirk Thatmann Sebastian Zickau Felix Beierle  “Distributed-Ledger-based Authentication with Decentralized Identifiers and Verifiable Credentials”  In _2nd Conference on Blockchain Research and Applications for Innovative Networks and Services_ IEEE, 2020
  * [5] J.L. Camp  “Digital Identity”  In _IEEE Technology and Society_ 24 IEEE, 2004
  * [6] Olivier Sanders David Pointcheval  “Short Randomizable Signatures”  In _The Cryptographers’ Track at the RSA Conference_ Springer, 2016
  * [7] “Decentralized Identifiers (DIDs) v1.0” Accessed: 2023-03-30, https://www.w3.org/TR/did-core/
  * [8] Amit Deo  “Variants of LWE: Attacks, Reductions, and a Construction”  In _Royal Holloway, University of London_ PhD Thesis Royal Holloway, University of London, 2019
  * [9] Jens Groth  “On the Size of Pairing-Based Non-interactive Arguments”  In _Annual International Conference on the Theory and Applications of Cryptographic Techniques_ Advances in Cryptology – EUROCRYPT 2016 Springer, 2016, pp. 305–326
  * [10] Fei Zhu Xun Yi Alsharif Abuadbba Junwei Lup Surya Nepal Xinyi Huang  “Efficient Hash-Based Redactable Signature for Smart Grid Applications”  In _European Symposium on Research in Computer Security_ 27 Association for Computing Machinery, 2022
  * [11] Yong Zhao Shaojun Yang Wei Wu Xinyi Huang  “A Lattice-Based Redactable Signature Scheme using Cryptographic Accumulators for Trees”  In _The Computer Journal_ Oxford University Press, 2022
  * [12] Justin Broaddus Anthony Skjellum Ilker Ozcelik Sai Medury  “An Overview of Cryptographic Accumulators”  In _Arxiv_ Arxiv, 2021
  * [13] Chen Yan Ji Fang  “Centralized Identity Authentication Research Based on Management Application Platform”  In _International Conference on Information Science and Engineering_ 1 IEEE, 2009
  * [14] Terrence Jo  “An Exploration of Zero-Knowledge Proofs and zk-SNARKs”  In _University of Pennsylvania_ University of Pennsylvania, 2019
  * [15] Ravneet Kaur  “Digital Signature”  In _International Conference on Computing Sciences_ IEEE, 2012
  * [16] Ali M Al-Khouri  “Digital identity: Transforming GCC economies”  In _Innovation: Organization and Management_ 16.2 Association for Computing Machinery, 2013
  * [17] Samia Bouzefrane Maryline Laurent  “Digital Identity Management”  Elsevier, 2015
  * [18] Hartwig Mayer  “zk-SNARK explained: Basic Principles”  In _Coinfabrik_ Coinfabrik, 2017
  * [19] Michael Rosenberg Jacob White Christina Garman Ian Miers  “zk-creds: Flexible Anonymous Credentials from zkSNARKs and Existing Identity Infrastructure”  In _Security and Privacy_ 2023 IEEE Symposium on Security and Privacy (SP) IEEE Computer Society, 2023, pp. 790–808
  * [20] Bertjan Doose Naomi Ellemers  “Self and Social Identity”  In _Annual Review of Psychology_ 53 Annual Reviews Inc., 2002, pp. 161–186
  * [21] Anca Nitulescu  “zk-SNARKs: A Gentle Introduction”  In _Ecole Normale Superieure_ Ecole Normale Superieure, 2020
  * [22] Yair Oren Oded Goldreich  “Definitions and properties of zero-knowledge proof systems”  In _Journal of Cryptology_ 7 Arxiv, 1994, pp. 1–32
  * [23] Khalifa Toumi Omar Dib  “Decentralized Identity Systems: Architecture, Challenges, Solutions and Future Directions”  In _Annals of Emerging Technologies in Computing_ 4.5 nternational Association of EducatorsResearchers, 2020
  * [24] Maksym Petkus  “Why and How zk-SNARK Works”  In _Arxiv_ Arxiv, 2019
  * [25] “RSS Code” Accessed: 2023-06-30, https://github.com/alan-turing-institute/RSS/tree/new_rss-vc-signing
  * [26] Olivier Sanders  “Improving Revocation for Group Signature with Redactable Signature”  In _IACR International Conference on Public-Key Cryptography_ PKC 2021 Springer, 2021, pp. 301–330
  * [27] Ian Miers Sean Bowe  “Scalable Multi-party Computation for zk-SNARK Parameters in the Random Beacon Model”  In _IACR Cryptology_ IACR, 2017
  * [28] “The BBS Signature Scheme” Accessed: 2023-03-30, https://identity.foundation/bbs-signature/draft-irtf-cfrg-bbs-signatures.html
  * [29] “The Zero Knowledge Blog” Accessed: 2023-03-30, https://www.zeroknowledgeblog.com
  * [30] Martin R. Valerio Cini Russell W.. Giulio Malavolta Sri AravindaKrishnan Thyagarajan  “Lattice-Based SNARKs: Publicly Verifiable, Preprocessing, and Recursively Composable”  In _Annual International Cryptology Conference_ Advances in Cryptology – CRYPTO 2022 Springer, 2022, pp. 102–132
  * [31] Markulf Kohlweiss Mary Maller Janno Siim Mikhail Volkhov  “Snarky Ceremonies”  In _International Conference on the Theory and Application of Cryptology and Information Security_ Advances in Cryptology – ASIACRYPT 2021 Springer, 2021, pp. 98–127

Beosin


# A Comparative Study of Deep Reinforcement Learning-based Transferable Energy
Management Strategies for Hybrid Electric Vehicles

Jingyi Xu1, Zirui Li2, Li Gao1, Junyi Ma1, Qi Liu1 and Yanan Zhao1,∗ This work
was supported by the National Key R&D Program of China under Grant
2018YFB0105205-02, Grant 2017YFC0804803 and Grant 2017YFC0804808.1 Jingyi Xu,
Li Gao, Junyi Ma, Qi Liu and Yanan Zhao are with School of Mechanical
Engineering, Beijing Institute of Technology, 100081 Beijing, China 2 Zirui Li
is with the Department of Transport and Planning, Delft University of
Technology. Delft 2628 CD, The Netherlands and is also with the School of
Mechanical Engineering, Beijing Institute of Technology, 100081 Beijing, China
* Corresponding author: Y. Zhao and Z. Li

###### Abstract

The deep reinforcement learning-based energy management strategies (EMS) have
become a promising solution for hybrid electric vehicles (HEVs). When driving
cycles are changed, the neural network will be retrained, which is a time-
consuming and laborious task. A more efficient way of choosing EMS is to
combine deep reinforcement learning (DRL) with transfer learning, which can
transfer knowledge of one domain to the other new domain, making the network
of the new domain reach convergence values quickly. Different exploration
methods of DRL, including adding action space noise and parameter space noise,
are compared against each other in the transfer learning process in this work.
Results indicate that the network added parameter space noise is more stable
and faster convergent than the others. In conclusion, the best exploration
method for transferable EMS is to add noise in the parameter space, while the
combination of action space noise and parameter space noise generally performs
poorly. Our code is available at https://github.com/BIT-XJY/RL-based-
Transferable-EMS.git.

##  I Introduction

Hybrid electric vehicles (HEVs) are currently important carriers of self-
driving technology [1]. HEVs involve two or more energy sources. Thus there is
a considerable need for energy management strategies (EMS) to distribute power
supplements among several power sources to improve energy efficiency and
reduce emissions [2]. There are mainly three types of EMS for HEVs: rule-based
methods, optimization-based techniques, and learning-based approaches [3].

The rule-based approach is the most common method to achieve real-time control
of HEVs, the effectiveness of which depends on the intuition and experience of
engineers [4]. To reduce the reliance on professional engineers, the
optimization-based method is introduced, using optimization algorithms to
solve for optimal or sub-optimal solutions in the feasible domain to obtain
better fuel economy [5]. According to different optimal control objectives and
algorithms, optimization-based EMS can be divided into global optimization and
real-time optimization approaches. The classical optimization-based methods
include linear programming algorithm (LP) [6], dynamic programming algorithm
(DP) [7, 8], equivalent fuel consumption minimization strategy (ECMS) [9, 10],
model predictive control (MPC) [11], etc. The above methods improve the real-
time performance and fuel economy of EMS to some extent, but they have more
computational cost than rule-based methods [12].

With the rapid development of machine learning in recent years, the learning-
based energy management method has become a promising solution for HEVs.
Current studies mainly focus on deep reinforcement learning (DRL) based EMS
due to their strong learning ability, where the EMS problem is modeled as a
Markov Decision Process (MDP). The optimal solution for EMS can be learned
through the interaction between agents and the environment. [13] used deep
Q-learning network (DQN) algorithm for energy management, which solved the
dimensional catastrophe problem. Based on this, [14] compared double deep
Q-learning with DQN for energy management of plug-in hybrid vehicles and
demonstrated advantages of the former in terms of convergence and fuel
economy. [15] showed that the energy management policy based on deep
deterministic policy gradient (DDPG) algorithm has a strong characterization
capability of deep neural networks and can improve fuel economy significantly.
In addition, [16] indicated that asynchronous advantage actor-critic (A3C) and
distributed proximal policy optimization (DPPO) improved the learning
efficiency.

Although DRL-based methods have made a significant breakthrough, their
limitations are the long training time for an agent to learn the optimal
solution through trial-and-error interactions with the environment [17].
Besides, the training process must be repeated even when encountering a new
but similar task. Therefore, some works have combined transfer learning with
DRL to improve the training efficiency between similar tasks. [18] combined
proximal policy optimization (PPO) and transfer learning to effectively reduce
time consumption and guarantee control performance. [19] combined DDPG and
transfer learning to derive an adaptive energy management controller for
hybrid tracked vehicles. Results show that this method has the potential to be
applied in real-world environments. [20] incorporated transfer learning into
DDPG-based EMS for HEVs to transfer knowledge among three types of HEVs that
have apparently different structures.

In DRL, the agent utilizes exploration methods to acquire knowledge about the
environment, which may explore better actions. The main approach is to add
different types of noise while selecting actions. Comparing the impact of
different exploration methods on DRL is implemented by much previous work [21,
22]. However, there are few studies considering the effects of exploration
methods on the combination of DRL and transfer learning, which improve the
training efficiency of algorithms and reduce the computational cost.

This work is a comparative study, which focuses on effects of different
exploration methods of DDPG for transferable EMS. DDPG combines advantages of
DQN and the actor-critic architecture, which leads to stability and high
efficiency. Thus, DDPG is appropriate for evaluating the strategies from
network parameters transferring. In this work, several types of noise are
added to DDPG netwoks which are trained by multiple driving cycles. Then,
training weights are saved to initialize a new DDPG network. The second
training process is performed with noise to acquire the optimal transferable
EMS.

In sum, the main contributions of this work are: Different types of noise,
i.e., action space noise and parameter space noise, are added to the DDPG
algorithm to explore in actions selection. Parameters of networks with
different exploration methods are used to initialize new networks. The methods
of exploration that work best for DDPG-based EMS and suit the most for
transfer learning in the training efficiency are given by the comparative
study.

The remainder of this work is organized as follows: Section II introduces the
proposed method in comparing effects of different exploration approaches of
DDPG-based EMS and the performance of the transferred new network; Section III
details experiment results, and the conclusion is depicted in Section IV.

##  II Deep Reinforcement Learning-based Transferable Energy Management
Strategies

A DRL-based transferable EMS is used to evaluate the performance of different
exploration methods. The sketch map of DRL-based transferable EMS is shown in
Fig.1. This section describes the HEV model, the DRL-based EMS formulation,
different types of noise added to DRL networks, and the effects of transferred
new domain networks using different kinds of noise. In this process, the
effects of different types of noise for exploration in DDPG and deep transfer
learning are compared in detail in Section III.

Figure 1: The sketch map of DRL-based transferable EMS.

###  II-A Hybrid Electric Vehicle Model

The EMS for Prius, one of the most classical HEVs, has been extensively
studied [23].

####  II-A1 Prius configuration

Prius is equipped with the Hybrid Synergy Drive system, which consists of an
internal combustion engine ICE, an electric motor MG2, and a generator MG1.
Prius is also equipped with a low-capacity nickel-metal hydride (Ni-MH)
battery used to drive the motor and generator. These systems in Fig.2 are
integrated with a power splitting planetary gear, which provides various power
flow configurations for different operating.

Figure 2: Architecture of Prius powertrain.

####  II-A2 Power request model

After building the Prius model, the vehicle power demand is calculated using
the longitudinal force balance equation. The longitudinal force $F$ consists
of rolling resistance $F_{f}$, aerodynamic drag $F_{w}$, gradient resistance
$F_{i}$, and inertial force $F_{a}$ [24]:

$\left\\{\begin{aligned} &F=F_{f}+F_{w}+F_{i}+F_{a}\\\ &F_{f}=mg\cdot f\\\
&F_{w}=\frac{1}{2}\rho\cdot A_{f}\cdot C_{D}\cdot v^{2}\\\ &F_{i}=mg\cdot i\\\
&F_{a}=m\cdot a\end{aligned}\right.$ (1)

where $m$ is the curb weight, $g$ is the gravitational constant, $f$ is the
rolling friction coefficient, $\rho$ is the air density, $A_{f}$ is the
fronted area, $C_{d}$ is the aerodynamic coefficient, $v$ is the speed in
regard to a certain driving cycle, $i$ is the road slope (not considered in
this paper), and $a$ is the acceleration.

####  II-A3 Powertrain system model

The engine, the electric motor, and the generator of the Prius are modeled by
their corresponding efficiency maps from bench tests. The Ni-MH battery is
modeled by an equivalent circuit model ignoring temperature changing and
battery aging:

$\left\\{\begin{aligned} &P(t)=I(t)\cdot V_{oc}(t)-R_{0}\cdot I^{2}(t)\\\
&I(t)=\frac{V_{oc}(t)-\sqrt{V^{2}_{oc}(t)-4\cdot R_{0}\cdot P(t)}}{2R_{0}}\\\
&SoC(t)=\frac{Q_{0}-\int_{0}^{t}I(t)dt}{Q}\end{aligned}\right.$ (2)

where $P$ is the output power, $I$ denotes the current, $V_{oc}$ is the open-
circuit voltage, $R_{0}$ is the internal resistance, $SoC$ is the state of
charge, $Q_{0}$ is the initial battery capacity, and $Q$ is the nominal
battery capacity. Details on Prius parameters are shown in Table I.

TABLE I: Parameters of Prius Components | Parameters | Values
---|---|---
Engine | Maximum power, $P_{e}$ | 56 kW
Maximum torque, $T_{e}$ | 120 Nm
Motor | Maximum power, $P_{m}$ | 50 kW
Maximum torque, $T_{m}$ | 400 Nm
Battery | Capacity, $Q$ | 1.54 kWh
Voltage, $V_{o}c$ | 237 V
Vehicle | Curb weight, $m$ | 1449 kg
Roll resistance coefficient, $f$ | 0.013
Air resistance coefficient, $f_{A}$ | 0.26
Frontal area, $A_{f}$ | 2.23 ${\rm m^{2}}$
Wheel radius, $r$ | 0.287 m
Transmission | Final gear ratio, $i_{g}$ | 3.93
Characteristic parameter, $C$ | 2.6

###  II-B DRL Formulation

A DRL problem that satisfies the Markov property can generally be modeled in
terms of the MDP, which can be characterized as
$(\mathbf{S},\mathbf{A},\mathbf{P},R,\gamma)$. $\mathbf{S}$ represents a set
of state spaces. $\mathbf{A}$ is a set of action spaces. $\mathbf{P}$ denotes
a state transition probability matrix. $R$ represents a reward function.
$\gamma$ denotes a discount factor.

DDPG is one of the most typical actor-critic DRL methods, which is an off-
policy and model-free algorithm. As illustrated in Fig.3, DDPG has an actor
network $\mu(\mathbf{s}|\theta^{\mu})$, a critic network
$Q(\mathbf{s},\mathbf{a}|\theta^{Q})$, an actor target network
$\mu^{\prime}(\mathbf{s}^{\prime}|\theta^{\mu^{\prime}})$, and a critic target
network
$Q^{\prime}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}|\theta^{Q^{\prime}})$. The
actor target network has the same structure as the actor network, while the
critic target network has the same structure as the critic network.
$\mathbf{s}$ is the agent state as the input of actor network and critic
network. $\mathbf{a}$ is the agent action as the output of actor network and
the input of critic network. $\theta$ represents parameters of the
corresponding network. $\mathbf{s}^{\prime}$ and $\mathbf{a}^{\prime}$ are
defined in the same way.

The DDPG algorithm is used to learn the optimal policy of Prius EMS in this
work. The neural network has a pyramid-like architecture, with the number of
neurons in hidden layers decreasing layer by layer. The optimal EMS utilizes
the DDPG algorithm, which is trained with different driving cycles.

Figure 3: The architecture of DDPG.

The DDPG-based EMS is formulated according to the following MDP.

####  II-B1 State space, $\mathbf{S}$

The state of the system,

$\mathbf{s}=\\{SoC,v,acc\\}$ (3)

which consists of $SoC$, the velocity of Prius $v$ and the acceleration $acc$.

####  II-B2 Action space, $\mathbf{A}$

At each episode, the agent can select actions in continuous engine power
$T_{eng}$:

$\mathbf{a}=\\{T_{eng}\\}$ (4)

####  II-B3 Reward function, $R$

There are two aspects of the reward function of DDPG-based EMS, the energy
consumption and the SoC sustaining. The multi-objective reward function is
defined as:

$r=-\\{\alpha[fuel(t)+elec(t)]+\beta[SoC_{ref}-SoC(t)]^{n}\\}$ (5)

where $\alpha$ is the weight of Prius consumption including the fuel
consumption of engine $fuel$ and the electricity consumption of motor $elec$,
$\beta$ is the weight of battery charge-sustaining, and $SoC_{ref}$ represents
the SoC reference value. The goal of the reward function is to minimize the
energy consumption and retain the battery SoC at an appropriate range. To
control for variables, in the following comparison, $SoC_{ref}$ is selected as
0.6 according to the minimum charge-discharge internal resistance. $\alpha$ is
selected as 1, $\beta$ is set to 350, and $n$ is set to 2, according to the
previous work [24].

###  II-C Transfer Learning

Traditional DRL algorithms are used to solve the problem with training and
test data in the same domain. However, once the domain is changed, the network
needs to be retrained, which is quite complex and time-consuming [25, 26, 27].
Transfer learning is extremely useful in solving this problem. When two
domains are similar, network parameters can be stored and reused in the new
one along with transfer learning approaches [28, 29, 30].

Given a source domain $M_{s}$ and a target domain $M_{t}$, transfer learning
aims to learn an optimal policy $\pi^{*}$ from $M_{s}$ for $M_{t}$. $M_{s}$
provides prior knowledge $D_{s}$ that is accessible for $M_{t}$. Thus, by
leveraging the information from $D_{s}$, the target agent learns better and
faster in $M_{t}$ [31].

A network that specializes in obtaining source EMS is used in our work. Since
driving cycles of $M_{s}$ and $M_{t}$ have the same feature space and are
correlated with each other, source domain knowledge can be transferred to the
novel, but relevant target domain [32]. The majority of parameters in the
neural network are the same, and only parameters of the output layer should be
retrained. Thus, both the source network and the target network use the same
DDPG architecture shown in Fig.3, and the weights of the source network except
for the last layer are used to initialize the target network that will be
trained on new driving cycles. Further details about hyperparameters of DDPG
in $M_{s}$ and $M_{t}$ are given in Table II.

In DDPG, the agent utilizes exploration to acquire knowledge about the
environment and applies exploitation to select a control action based on
current knowledge [33]. Thus, the coordination between exploitation and
exploration is essential. The following parts of this subsection provide a
description of exploration methods used in the DDPG-based transferable EMS.

The primary purpose of exploration is to avoid local optimum for agent’s
behaviors [21]. Thus, to realize efficient and effective exploration, random
noise is frequently added to perturb selected actions, which is mainly focused
on in our work. Main approaches can be classified into two groups: adding
noise in the action space and adding noise directly to agent’s parameters.

####  II-C1 Action space noise

When the agent selects actions using the actor network, the noise
$\mathcal{N}$ is added to the action space. The final selected action $a_{t}$
at each step satisfies:

$a_{t}=\mu(s|\theta^{\mu})+\mathcal{N}$ (6)

Action space noise could be a simple Gaussian noise or a more advanced
Ornstein-Uhlenbeck (OU) correlated noise process [22]. Gaussian noise
satisfies $\mathcal{N}\sim N(0,\sigma^{2}I)$, where $\sigma^{2}$ denotes
variance and the expected value is set to 0. An OU process [34] can be used as
a temporally correlated noise. Just like the Gaussian noise mentioned above,
the expected value of OU noise $\mathcal{N}\sim OU(0,\sigma^{2})$ is set to 0,
and the variance can be set to multiple values.

####  II-C2 Parameter space noise

While adding noise in the action space to explore, there is no guarantee that
the same action will be chosen in the same state each time, which can lead to
inconsistent exploration. The parameter space noise solves this problem and
directly perturbs actor network parameters to get a rich set of behaviors. The
final selected action $a_{t}$ at each step satisfies:

$\left\\{\begin{aligned} &a_{t}=\mu(s|\tilde{\theta}^{\mu})\\\
&\tilde{\theta}=\theta+N(0,\sigma^{2}I)\end{aligned}\right.$ (7)

TABLE II: DDPG Hyperparameters Parameters | Source Domain | Target Domain
---|---|---
Number of Episodes, $K$ | 1000 | 300
Replay memory size, $M$ | 50000 | 50000
Learning rate of actor network, $lr_{a}$ | 0.001 | 0.0009
Learning rate of critic network, $lr_{c}$ | 0.01 | 0.009
Discount factor, $\gamma$ | 0.9 | 0.9
Target network update frequency, $\tau$ | 0.01 | 0.01
Mini-batch size, $batch$ | 64 | 64

##  III Experiments

The purpose of our work is to compare effects of different exploration methods
of DDPG-based EMS and transferred new networks in terms of transfer
efficiency. Driving cycles are selected for the source domain and the target
domain, which are different but similar. Then, networks with different
exploration methods are trained in the source domain, of which parameters are
saved. Finally, the adaptation of target domain networks, of which parameters
are initialized using saved weights, is evaluated.

###  III-A Driving Cycles

In this work, driving cycles are all selected from standard data. Source tasks
are performed over multiple cycles, including Urban Dynamometer Driving
Schedule (UDDS) [35], FTP75 [24], etc. Target tasks are conducted on New
European Driving Cycle (NEDC) [36], which is different from driving cycles
used in the source domain but similar. Using multiple driving cycles for
training in the source domain improves the generalization ability of the
trained model, which leads to better transfer results. A driving cycle with a
high similarity to the source driving cycles is chosen for the target domain,
since similarity is a necessary factor for transfer learning.

###  III-B Training in the Source Domain

To ensure the validity of weights to be transferred, networks with different
exploration methods are firstly trained on the source domain. Settings of
networks with different types of noise are shown in Table III. By comparative
studying, suitable networks, of which parameters are used to initiate weights
of target networks, are selected according to the training results.

TABLE III: Networks added different types of noise Space Added Noise | Noise Type | Variance
---|---|---
Action space | Gaussian | 0.02,0.03,0.04,0.05,0.06
OU | 0.08,0.09,0.10,0.11,0.13
Parameter space | Gaussian |  0.03,0.04
Action & parameter space | Gaussian & Gaussian | 0.06 & 0.03
OU & Gaussian | 0.09 & 0.03

(a) Action space noise with Gaussian

(b) Action space noise with OU

(c) Parameter space noise with Gaussian

(d) Mixture noise

(e) Transferred networks

Figure 4: Comparison of different exploration methods in the source domain.

As described in Fig.4, Gaussian noise added in the action space, OU noise
added in the action space, Gaussian noise added in the parameter space, and
their mixture are used to explore in DDPG-based EMS. In Fig.4(a), different
variance values $\sigma^{2}$ of Gaussian noise added in the action space are
set to 0.02, 0.03, 0.04, 0.05, 0.06, respectively. The reward fluctuates the
most when the variance is set to 0.02. Only the network with variance 0.05
shows considerable oscillations once the trained weights converge. Thus, the
network with variance 0.06 is the most stable. Similarly, Fig.4(b) and 4(c)
illustrate that the network’s variance of 0.09 with OU noise added in the
action space and the network’s variance of 0.03 with Gaussian noise added in
the parameter space are the most stable, respectively. Results of other noise
configurations, which converge to a local optimum or fluctuate too much, are
not shown here. Besides, different effects of a single noise and a mixed noise
are compared in Fig.4(d). Results using the Gaussian noise added in the action
space and parameter space are very unstable, as shown by the yellow line. The
noise of multiple Gaussian processes makes the agent tend to explore rather
than exploit to a great extent, leading to more non-optimal actions with
fluctuating reward values. The most stable is the Gaussian noise added in the
action space, followed by the Gaussian noise added in both the action space
and parameter space.

Above all, the most stable network is the one that utilizes the Gaussian noise
added in the action space with 0.06 variance to explore, followed by the
network with the combination of OU noise added in the action space and
Gaussian noise added in the parameter space to explore. Chosen networks which
are transferred to a new domain are shown in Fig.4(e).

###  III-C Adaptation of Transfer Learning

Results of the transferred DDPG in the target domain using different
exploration methods in EMS are discussed in this subsection. A new network is
trained on the new driving cycle, learning from scratch or initializing the
network parameters using prior ones.

TABLE IV: Mean reward and Iteration Number of Target Network | Exploration Method
---
(Target Network)
| Transferred Network Parameter
---
(Source Network)
| Mean Return
---
(First 50 Episodes)
| Mean Return
---
(Convergence Interval)
| Iteration
---
Number
No noise | TFS | -21.6386 | -1.2101 | 30
Gaussian_AS | -0.9186 | -1.0540 | 20
OU_AS | -0.9451 | -1.2019 | 29
Gaussian_PS | -0.9009 | -1.4133 | 21
APS | -2.0715 | -1.1878 | 29
Gaussian noise added in the ation space | TFS | -20.9453 | -0.8766 | 36
Gaussian_AS | -0.8780 | -0.8776 | 35
OU_AS | -0.9748 | -0.8908 | 30
Gaussian_PS | -0.7987 | -0.9071 | 25
APS | -1.7628 | -0.9159 | 32
OU noise added in the ation space | TFS | -25.3330 | -1.1527 | 36
Gaussian_AS | -1.1239 | -1.0585 | 24
OU_AS | -0.9035 | -1.0417 | 22
Gaussian_PS | -0.8720 | -1.1611 | 31
APS | -2.3262 | -1.0474 | 23
Gaussian noise added in the parameter space | TFS | -21.6009 | -1.1523 | 35
Gaussian_AS | -1.0250 | -1.1132 | 21
OU_AS | -1.1109 | -1.1652 | 22
Gaussian_PS | -1.5669 | -1.2466 | 34
APS | -4.4550 | -1.2734 | 30
Noise added in ation space and parameter space | TFS | -21.1288 | -1.0919 | 35
Gaussian_AS | -1.4819 | -1.0066 | 31
OU_AS | -0.9408 | -1.1473 | 27
Gaussian_PS | -13.8613 | -0.9998 | 24
APS | -2.6120 | -1.1589 | 30

As shown in Table IV, following criteria are adopted to evaluate the
adaptation of the transferable EMS [37]:

####  III-C1 Jumpstart Performance (JP)

The initial performance of the agent. The mean reward of the first 50 episodes
is used to evaluate it.

####  III-C2 Asymptotic Performance (AP)

The ultimate performance of the agent. The mean reward of the convergence
interval is used to evaluate it.

####  III-C3 Time to Threshold (TT)

The learning time needed for the target agent to reach a certain performance
threshold. The iteration number of convergence is used to evaluate it.

(a) No noise

(b) Gaussian action space noise

(c) OU action space noise

(d) Parameter space noise

(e) Action space and parameter space noise

Figure 5: Comparison of different exploration methods in the target domain.

In Table IV, TFS means the source network trained from scratch. Gaussian_AS,
OU_AS, Gaussian_PS and APS mean source networks with Gaussian noise added in
the action space, OU noise added in the action space, Gaussian noise added in
the parameter space, and both OU noise added in the action space and Gaussian
noise added in the parameter space, respectively. While target networks use
the same type of noise, the maximum reward value obtained from first 50
episodes is larger than the maximum convergence value in the convergence
interval, which means that the DDPG network falls into a local optimum during
the exploration process. Besides, the initialization of parameters of target
networks with parameter space noise generally works well in terms of
convergence speed and mean reward value, except for target networks using the
mixture of action space and parameter space noise for exploration. After
target networks have converged, convergence values of different exploration
methods do not differ significantly.

As shown in Fig.5(a), no noise is added on the target network. In first 50
episodes, the network which is trained from scratch (gray line) starts out
with a small reward value, which means that its JP is poor. The green, blue,
yellow, and orange lines represent parameters of networks with different
exploration methods to initialize current target networks. Weights of these
networks are perturbed by Gaussian noise added in the action space
($\sigma^{2}=0.06$), OU noise added in the action space ($\sigma^{2}=0.09$),
Gaussian noise added in the parameter space ($\sigma^{2}=0.03$), both OU noise
added in the action space noise and Gaussian noise added in the parameter
space to explore, respectively. The yellow line fluctuates the least. In the
convergence interval (50 300 episodes in all target cases), the green line is
the most stable, gray and blue lines are the second most unstable, and the
most unstable one is the yellow line. In general, they do not differ much.
This means that it is the most stable to initialize the target network with
parameters of the original network with Gaussian action space noise.

The Gaussian noise is added in the action space of target networks, of which
the training process is shown in Fig.5(b). In first 50 episodes, the gray line
fluctuates the most, and the orange line fluctuates the least, while the blue
line fluctuates the least in the convergence interval. This indicates that the
most stable approach is to initialize the target network with parameters of
the original network with OU action space noise, and use Gaussian action space
noise to explore.

The action space noise with OU process is added on the target network, of
which the training process is shown in Fig.5(c). The gray line has the most
considerable fluctuation, followed by the orange line in first 50 episodes.
The blue line has the slightest fluctuation. In the convergence interval, the
most stable one is still the blue line, which indicates that it has a better
learning effect to initialize the target network, which uses OU action space
noise to explore, with parameters of the original network with OU action space
noise.

The Gaussian noise added in the parameter space is added on the target
network, of which the training process is shown in Fig.5(d). In first 50
episodes, the gray line has a small initial value because it does not have any
prior knowledge. The orange line fluctuates the second most, which means that
parameters of the network with action space noise and parameter space noise
are not suitable for initializing the new target network with parameter space
noise. In the convergence interval, the most stable one is still the blue
line, followed by the green line, and the most unstable one is the yellow
line. The blue line is the most stable one throughout the training process.

Both the action space noise and the parameter space noise are added on the
target network, of which the training process is shown in Fig.5(e). The gray
line has the smallest initial value, while the yellow line has the largest
fluctuation in first 50 episodes. It means that parameters of the network with
parameter space noise are not suitable for initializing the target network
with action space noise and parameter space noise. In the convergence
interval, the fluctuation of each method is negligible.

Combining results expressed in Fig.4 and Fig.5, the network with the action
space noise works best for DDPG-based EMS. For transfer learning, the network
with the parameter space noise is the most stable, while the network with
multiple noises of action space and parameter space has poor initial
performance.

##  IV Conclusion

In this paper, choosing an optimal and efficient EMS is formulated as a deep
reinforcement learning-based transfer learning problem. We compared different
exploration approaches for deep reinforcement learning and transfer learning
to find out the best energy management strategy. Effects of action space noise
and parameter space noise which are added to the DDPG algorithm, are
presented. Experimental results show that the method of parameter space noise
exploration works best for DDPG-based transferable EMS.

Despite these encouraging comparison results, there are several avenues for
future research. First, we want to investigate other exploration modalities,
not only adding noise for action selecting. We furthermore plan to compare
different exploration methods based on more robust DRL algorithms, such as
twin delayed DDPG and soft actor-critic.

## References

  * [1] T. Liu, W. Tan, X. Tang, J. Zhang, Y. Xing, and D. Cao, “Driving conditions-driven energy management strategies for hybrid electric vehicles: A review,” _Renewable and Sustainable Energy Reviews_ , vol. 151, p. 111521, 2021.
  * [2] T. Liu and X. Hu, “A bi-level control for energy efficiency improvement of a hybrid tracked vehicle,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 4, pp. 1616–1625, 2018.
  * [3] X. Guo, T. Liu, B. Tang, X. Tang, J. Zhang, W. Tan, and S. Jin, “Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle,” _IEEE Access_ , vol. 8, pp. 165 837–165 848, 2020.
  * [4] H. Guo, X. Wang, and L. Li, “State-of-charge-constraint-based energy management strategy of plug-in hybrid electric vehicle with bus route,” _Energy Conversion and Management_ , vol. 199, p. 111972, 2019.
  * [5] T. Hofman, M. Steinbuch, R. Van Druten, and A. Serrarens, “Rule-based energy management strategies for hybrid vehicles,” _International Journal of Electric and Hybrid Vehicles_ , vol. 1, no. 1, pp. 71–94, 2007.
  * [6] F. Z. Kadda, S. Zouggar, and M. L. Elhafyani, “Optimal energy management of an autonomous hybrid system by using the linear programming method,” in _2013 International Renewable and Sustainable Energy Conference (IRSEC)_. IEEE, 2013, pp. 420–425.
  * [7] C.-C. Lin, H. Peng, J. W. Grizzle, and J.-M. Kang, “Power management strategy for a parallel hybrid electric truck,” _IEEE transactions on control systems technology_ , vol. 11, no. 6, pp. 839–849, 2003.
  * [8] X. Wang, H. He, F. Sun, and J. Zhang, “Application study on the dynamic programming algorithm for energy management of plug-in hybrid electric vehicles,” _Energies_ , vol. 8, no. 4, pp. 3225–3244, 2015.
  * [9] A. Rezaei, J. B. Burl, and B. Zhou, “Estimation of the ecms equivalent factor bounds for hybrid electric vehicles,” _IEEE Transactions on Control Systems Technology_ , vol. 26, no. 6, pp. 2198–2205, 2017.
  * [10] Q. Jiang, F. Ossart, and C. Marchand, “Comparative study of real-time hev energy management strategies,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 12, pp. 10 875–10 888, 2017.
  * [11] H. Borhan, A. Vahidi, A. M. Phillips, M. L. Kuang, I. V. Kolmanovsky, and S. Di Cairano, “Mpc-based energy management of a power-split hybrid electric vehicle,” _IEEE Transactions on Control Systems Technology_ , vol. 20, no. 3, pp. 593–603, 2011.
  * [12] R. Johri and Z. Filipi, “Optimal energy management of a series hybrid vehicle with combined fuel economy and low-emission objectives,” _Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering_ , vol. 228, no. 12, pp. 1424–1439, 2014.
  * [13] J. Wu, H. He, J. Peng, Y. Li, and Z. Li, “Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus,” _Applied energy_ , vol. 222, pp. 799–811, 2018.
  * [14] X. Qi, Y. Luo, G. Wu, K. Boriboonsomsin, and M. Barth, “Deep reinforcement learning enabled self-learning control for energy efficient driving,” _Transportation Research Part C: Emerging Technologies_ , vol. 99, pp. 67–81, 2019.
  * [15] Y. Wu, H. Tan, J. Peng, H. Zhang, and H. He, “Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus,” _Applied energy_ , vol. 247, pp. 454–466, 2019.
  * [16] X. Tang, J. Chen, T. Liu, Y. Qin, and D. Cao, “Distributed deep reinforcement learning-based energy and emission management strategy for hybrid electric vehicles,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 10, pp. 9922–9934, 2021.
  * [17] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 26–38, 2017.
  * [18] T. Liu, B. Wang, W. Tan, S. Lu, and Y. Yang, “Data-driven transferred energy management strategy for hybrid electric vehicles via deep reinforcement learning,” _arXiv preprint arXiv:2009.03289_ , 2020.
  * [19] X. Guo, T. Liu, B. Tang, X. Tang, J. Zhang, W. Tan, and S. Jin, “Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle,” _IEEE Access_ , vol. 8, pp. 165 837–165 848, 2020.
  * [20] R. Lian, H. Tan, J. Peng, Q. Li, and Y. Wu, “Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 8, pp. 8367–8380, 2020.
  * [21] M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz, “Parameter space noise for exploration,” _arXiv preprint arXiv:1706.01905_ , 2017.
  * [22] C. Colas, O. Sigaud, and P.-Y. Oudeyer, “Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms,” in _International conference on machine learning_. PMLR, 2018, pp. 1039–1048.
  * [23] D. V. Prokhorov, “Toyota prius hev neurocontrol and diagnostics,” _Neural Networks_ , vol. 21, no. 2-3, pp. 458–465, 2008.
  * [24] R. Lian, J. Peng, Y. Wu, H. Tan, and H. Zhang, “Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle,” _Energy_ , vol. 197, p. 117297, 2020.
  * [25] C. Lu, F. Hu, D. Cao, J. Gong, Y. Xing, and Z. Li, “Virtual-to-real knowledge transfer for driving behavior recognition: Framework and a case study,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 7, pp. 6391–6402, 2019.
  * [26] Z. Li, C. Gong, C. Lu, J. Gong, J. Lu, Y. Xu, and F. Hu, “Transferable driver behavior learning via distribution adaption in the lane change scenario,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 193–200.
  * [27] C. Gong, Z. Li, C. Lu, J. Gong, and F. Hu, “A comparative study on transferable driver behavior learning methods in the lane-changing scenario,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 3999–4005.
  * [28] Z. Li, J. Gong, C. Lu, and J. Li, “Personalized driver braking behavior modelling in the car-following scenario: An importance weight-based transfer learning approach,” _IEEE Transactions on Industrial Electronics_ , 2022\.
  * [29] Z. Li, J. Gong, C. Lu, and J. Xi, “Importance weighted gaussian process regression for transferable driver behaviour learning in the lane change scenario,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 11, pp. 12 497–12 509, 2020.
  * [30] C. Lu, F. Hu, D. Cao, J. Gong, Y. Xing, and Z. Li, “Transfer learning for driver model adaptation in lane-changing scenarios using manifold alignment,” _IEEE transactions on intelligent transportation systems_ , vol. 21, no. 8, pp. 3281–3293, 2019.
  * [31] Z. Zhu, K. Lin, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” _arXiv preprint arXiv:2009.07888_ , 2020.
  * [32] X. Guo, T. Liu, B. Tang, X. Tang, J. Zhang, W. Tan, and S. Jin, “Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle,” _IEEE Access_ , vol. 8, pp. 165 837–165 848, 2020.
  * [33] X. Hu, T. Liu, X. Qi, and M. Barth, “Reinforcement learning for hybrid and plug-in hybrid electric vehicle energy management: Recent advances and prospects,” _IEEE Industrial Electronics Magazine_ , vol. 13, no. 3, pp. 16–25, 2019.
  * [34] G. E. Uhlenbeck and L. S. Ornstein, “On the theory of the brownian motion,” _Physical review_ , vol. 36, no. 5, p. 823, 1930.
  * [35] N. Yang, L. Han, C. Xiang, H. Liu, and X. Hou, “Energy management for a hybrid electric vehicle based on blended reinforcement learning with backward focusing and prioritized sweeping,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 4, pp. 3136–3148, 2021.
  * [36] T. Liu, X. Hu, W. Hu, and Y. Zou, “A heuristic planning reinforcement learning-based energy management for power-split plug-in hybrid electric vehicles,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 12, pp. 6436–6445, 2019.
  * [37] Z. Zhu, K. Lin, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” _arXiv preprint arXiv:2009.07888_ , 2020.