Yikang Wang1,2, Hiromitsu Nishizaki1, Ming Li2,3

# Pretraining Conformer with ASR or ASV for Anti-Spoofing Countermeasure

###### Abstract

Finding synthetic artifacts of spoofing data will help the anti-spoofing
countermeasures (CMs) system discriminate between spoofed and real speech. The
Conformer combines the best of convolutional neural network and the
Transformer, allowing it to aggregate global and local information. This may
benefit the CM system to capture the synthetic artifacts hidden both locally
and globally. In this paper, we present the transfer learning based MFA-
Conformer structure for CM systems. By pre-training the Conformer encoder with
different tasks, the robustness of the CM system is enhanced. The proposed
method is evaluated on both Chinese and English spoofing detection databases.
In the FAD clean set, proposed method achieves an EER of 0.04%, which
dramatically outperforms the baseline. Our system is also comparable to the
pre-training methods base on Wav2Vec 2.0. Moreover, we also provide a detailed
analysis of the robustness of different models.

Index Terms: Anti-spoofing countermeasure system, fake audio detection,
transfer learning, conformer model, robustness

##  1 Introduction

Automatic speaker verification (ASV) techniques have become widely used in
real-life due to the development of deep neural network [1, 2, 3]. However,
the rapid development of generative techniques, e.g. text-to-speech synthesis
(TTS) and voice conversion (VC), makes the ASV systems vulnerable [4, 5].
Hense, a robust anti-spoofing countermeasures (CMs) system, is very important
as a safeguard for the ASV systems [6, 7].

There are two major challenges in logical access (LA) based anti-spoofing CM
tasks [8]. One is the noise robustness problem. In practice, end devices often
collect data in complex scenarios that contain a lot of noise, which leads to
degraded performance of the CM systems. Another is the problem of unseen
spoofing data detection. As the TTS and VC algorithms continue to advance, the
CM system has to face unseen data generated by unknown spoofing algorithms,
which also leads to low accuracy. In this paper, besides using the English-
based ASVspoof database [9, 10], we also focus on the FAD database [11], which
consists of Chinese speech data, and separating seen and unseen samples as
different subsets in the test set. In this work, we want to build a rubust CM
system in noisy and unseen data scenarios.

Recent research works have shown that self-supervised learning (SSL) using
large models can learn generalized speech representations from vast amounts of
unlabeled data, demonstrating robustness and strong generalization in various
speech-related downstream tasks [12]. Models such as Wav2Vec [13], HuBERT
[14], Wav2Vec 2.0 [15, 16], and WavLM [17] have exhibited promising results in
speech recognition [15], emotion recognition [18], speaker recognition [3],
and also anti-spoofing CM tasks [19, 20]. Wang et al. [19], compare the
performance of CM systems with different combinations of self-supervised
pretrained front-ends and various back-ends. Tak et al. [20] discusse the
potential to improve generalization and domain robustness through the use of
wav2vec 2.0 XLSR as front-end of AASIST [21] CM network. They also suggest
using telephone channel based data augmentation techniques, such as Rawboost
[22], to enhance the model robustness. Lee et al. [23] investigate the
exposure of synthetic artifacts in different feature spaces by taking out the
outputs of different layers in a 24-layer Transformer front-end as features.
However, training or fine-tuning models with over 300 million parameters is
extremely time-consuming and requires large scale computing resources. The
MFA-Conformer model effectively integrates global and local information [24,
25], and has great potential to be used in robust ASV or anti-spoofing CM
tasks. Since the Conformer model has been widely used for ASR , we can easily
perform transfer learning on ASR models pre-trained on large amount of data
[26]. In addition, the Conformer model has a much smaller model size compared
to other front-end big models, e.g. WavLM [17].

Figure 1: The proposed transfer learning based MFA-Conformer structure for CM
systems.

In this paper, we propose a transfer learning based MFA-Conformer structure
for CM systems. We first pre-train the MFA-conformer model in the ASR or ASV
task, and then trim the model to obtain the encoder part, which is fine-tuned
on different anti-spoofing databases. Experimental results show that the MFA-
Conformer model trained using the proposed transfer learning method achieves
better detection performance in both seen and unseen conditions compared to
other models in the control group. The main contributions of this paper are
summarized as follows:

  * •

We introduce the MFA-Conformer model into the anti-spoofing CM task. Results
demonstrate the effectiveness of transfer learning with ASR or ASV pre-trained
models.

  * •

We analyze the robustness of different CM models against specific spoofing
algorithms, propose the error-prone tendency (ET) as a judging metric, and
visualize the results in line graphs. This may help the choice of model fusion
or help in selecting appropriate features for spoofing algorithm traceability
tasks.

##  2 Method

###  2.1 Conformer architecture

This section describes the main structure of the Conformer encoder used for
ASR [24], as well as the MFA-Conformer for ASV tasks [25]. In this paper, the
original Conformer and MFA-Conformer structures are pre-trained in the ASR and
ASV tasks, respectively, and subsequently fine-tuned for different databases
after removing the redundant parts of the model and connecting its encoder to
the backend as a trainable feature extractor as shown in Figure 1.

####  2.1.1 Conformer encoder for ASR

To learn both position-wise local features and content-based global
interactions, Gulati et al. [24] proposed the Conformer model. In this
network, the input audio signals are processed through feature extractor and a
subsampling convolution layer and then fed into the Conformer encoder composed
of multiple Conformer blocks. A Conformer block is a sandwich structure of
four modules stacked together, i.e., a feed-forward (FFN) module, a multi-
headed self-attention module (MHSA), a convolution module, and a second FFN
module in the end.

The MHSA is employed with a relative sinusoidal positional encoding scheme
from Transformer XL [27] to improve generalization on varying input length.
The subsequent convolution module contains a point-wise convolution and a
gated linear unit (GLU) activation layer, followed by a single 1-D depth-wise
convolution layer with batchnorm to training deeper models. Two half-step FFN
layers replacing the original FFN layer in the Transformer block, one before
the attention layer and one after. In general [24], this structure can be
represented as

$\displaystyle\tilde{x_{i}}$
$\displaystyle=x_{i}+\frac{1}{2}\operatorname{FFN}\left(x_{i}\right)$ (1)
$\displaystyle x_{i}^{\prime}$
$\displaystyle=\tilde{x_{i}}+\operatorname{MHSA}\left(\tilde{x_{i}}\right)$
(2) $\displaystyle x_{i}^{\prime\prime}$
$\displaystyle=x_{i}^{\prime}+\operatorname{Conv}\left(x_{i}^{\prime}\right)$
(3) $\displaystyle y_{i}$
$\displaystyle=\operatorname{Layernorm}\left(x_{i}^{\prime\prime}+\frac{1}{2}\operatorname{FFN}\left(x_{i}^{\prime\prime}\right)\right)$
(4)

where FFN refers to the feed forward module, MHSA refers to the Multi-Head
Self-Attention module, and Conv refers to the convolution module.

####  2.1.2 MFA-Conformer encoder for ASV

Chen et al. [3] found that superimposing the output of each Transformer block
after hidden layers during pretraining can produce better representations for
speaker recognition tasks compared to simply using the output of the last
Transformer block layer. Similarly, Zhang et al. [25] verified this conclusion
on the Conformer model and proposed the Multi-scale Feature Aggregation
Conformer (MFA-Conformer). This network integrates information from multiple
Conformer blocks and connects the outputs of multiple scales, which are then
normalized and pooled together to obtain speaker embeddings. In short, the
MFA-Conformer is a concatenation of the output of each layer of the original
Conformer, leaving the construction of each layer unchanged.

###  2.2 Transfer learning strategy

For the Conformer pretrained models with two different target tasks, we adopt
two subtly different transfer learning strategy. Specifically, we load the
parameters of the pretrained Conformer encoder into the MFA-Conformer encoder
of our CM system, the output of each Conformer block is concatenated to
extract embeddings, and an attentive statistics pooling (ASP) layer and a
fully-connected (FC) layers are connected to obtain segment-level features,
then a linear classifier is connected at the back-end for fine-tuning. The
overall system structure is shown in Figure 1. During the transfer learning
based fine-tuning process, we do not always freeze the MFA-Conformer as a pure
front-end feature extractor; instead, we first fix the parameters of the
Conformer encoder and update the parameters for a few training epochs. Then,
we jointly fine-tune the Conformer encoder and the linear layer classifier as
a whole during training for the anti-spoofing CM tasks.

##  3 Experimental setup

###  3.1 Data preparation

The contents of the databases used in this paper are summarized in Table 1.
The fake audio detection (FAD) database [25] is a Chinese-mandarin database
for anti-spoofing CM tasks. It was built to investigate the robustness of
spoofing detection methods under noisy conditions. The FAD database has two
versions: a clean version and a noisy version. Both versions are divided into
different training, development, and test sets in the same way, with no
overlap of speakers between the three subsets. Each test set is further
divided into seen and unseen subsets. The unseen subset can evaluate the
generalization of the CM system to unknown spoofing methods. For data
augmentation of experiments on FAD database, we used an on-the-fly data
augmentation method [28], which is more diverse and efficient. Specifically,
when loading audio data, we randomly selected 2/3 of the data in each
minibatch for noise addition. The data augmentation method includes adding
background noise (environmental noise, music, babble noise) and adding
convolutional reverberation. The two augmentation methods use the MUSAN
database [29] and the room impulse responses (RIR) database [30],
respectively. During the on-the-fly data augmentation, the method is randomly
selected, and the signal-to-noise ratio is randomly set in the 0 to 20 dB
range.

The ASVspoof database is a series of data from the ASVspoof challenges [31,
32, 9, 10]. For the experiments, we use the ASVspoof 2019 LA (19LA) [9] and
the ASVspoof 2021 LA (21LA) [10] datasets. The 19LA dataset was created using
utterances from 107 speakers (46 male, 61 female). The set of 107 speakers is
partitioned into three speaker-disjoint sets for training, development, and
evaluation. The spoofed utterances were generated using four TTS and two VC
algorithms in the training and development sets, while 13 TTS/VC algorithms
are used in the evaluation set, 4 of which are partial-known and 7 of which
are unknown for training and development. The 21LA dataset remains the
training and development data unchanged and only proposes a new evaluation set
that contains attacks using the same simulation methods as the 19LA evaluation
set. The 21LA evaluation set consists of the data in various telephone
transmission systems, including Voice over Internet Protocol (VoIP) and the
Public Switched Telephone Networks (PSTN), thus exhibiting real-world signal
transmission channel effects. The experiments using ASVspoof database in this
paper are conducted using the 19LA train as the training set, and the three
models with the lowest loss in the 19LA development set are selected to be
tested in the respective evaluation sets of 19LA and 21LA dataset. Since the
19LA dataset consists of clean data, for the fairness of the comparison, none
of our experiments in the ASVspoof database use any data augmentation in fine-
turning CM models.

Table 1: The data distribution of each database.

| ASVspoof 2019 LA | ASVspoof 2021 LA
---|---|---
train | dev | evaluation | evaluation
bona fide | 2,580 | 2,580 | 7,355 | 14,816
spoof | 22,800 | 22,296 | 63,882 | 133,360
| FAD clean/noisy data
| train | dev | test seen | test unseen
bona fide | 12,800 | 4,800 | 14,000 | 7,000
spoof | 25,600 | 9,600 | 28,000 | 14,000

###  3.2 Model pretraining

####  3.2.1 Pretrained ASR Conformer model

We use the NEMO STT En Conformer-CTC Small model version 1.0.0 as the
pretrained ASR model
111https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/
stt_en_conformer_ctc_small, which has the same model structure as [24] but
replaces the Conformer transducer in [24] with a linear decoder backend and
links a connectionist temporal classification (CTC) for decoding. According to
the open source code
222https://github.com/NVIDIA/NeMo/blob/2ef544ffe6daa80d38d
0f494a7e42adcac50a4b9/examples/asr/conf/ssl/conformer/conformer_ssl.yaml, the
convolutional layer of the Conformer-CTC small model has a downsampling rate
of 1/4. The encoder part has 13M parameters with 4 attention heads and 16
conformer blocks. The feature dimension of the encoder convolutional layer is
176, and the feature dimension of the FFN is 704. The model was trained on a
composite database called NeMo ASRSET 1.4.1, which comprises more than 34,000
hours of English speech [26, 33].

####  3.2.2 ASV Conformer pretraining

For the pre-training of MFA-Conformer with ASV as the target task, we adopt
the identical Conformer encoder construction as the NEMO encoder. However, we
concatenate the output of each Conformer blocks to create the output
embedding. The development set of VoxCeleb 2 [34] is utilized to train the
model from scratch. The training data is collected from 5,994 speakers and
contains 1,092,009 speech files. To augment the training data, we employ
velocity perturbation techniques, which involve modifying the original
database. Specifically, we accelerate the speech by a factor of 1.1 and
decelerate them by a factor of 0.9, resulting in two additional versions of
each recording. Consequently, the training dataset expand to include 17,982
speakers and a total of 3,276,027 utterances. During the ASV model training,
we refer to the hyperparameter settings of Cai et al. [26] and finally obtain
speaker verification results on VoxCeleb 1 consistent with its description in
the article.

###  3.3 Network setup and evaluation metrics

We used log Mel-Filter Bank energy (FBANK) as the acoustic feature in
Conformer based experiments. We extract Fast Fourier Transform (FFT)
spectrograms with a window length of 1024 and a hop length of 128, using
Blackman windows. The number of Mel-filters in FBANK is set to 80 dimensions.

During fine-tuning on the FAD database, speech samples are truncated or
repeated up to 8 seconds before being loaded into the network for CM task
fine-tuning. We used Cross-Entropy softmax as the loss function. AdamW is used
as the optimizer with an initial learning rate of 0.001. We apply a cosine
annealing learning rate scheduler and a 4000-step warm-up strategy. Each
experiment is performed using one NVIDIA RTX A6000 GPU, and we set the batch
size to 256, with 100 epochs of training per model.

While fine-tuning on the ASVspoof databases, speech samples are truncated or
repeated up to 5 seconds. We then reduce the batch size to 64, set the initial
learning rate to 0.0001, and increase the dropout by 50% in the FC layer after
the ASP to prevent model overfitting. For the control group model, we use the
code and hyperparameter settings mentioned in [11, 35, 20]. The final
evaluation results are averaged over the selected epochs for each model, which
are obtained from the epochs in the development subset with the top-3 lowest
loss.

Table 2: The EERs (%) of each CM system trained with different database on
different evaluation set. The best performance among Conformer models is shown
in italics, and the best performance among all models is shown in bold. The
performance is reported as ``average(best)'' from the Top-3 models. $\dagger$
Note that RawBoost was not utilized in all three databases when reproducing
the W2V-AASIST models, as telephone transmission coding was involved in the
FAD database and 19LA dataset.

Model | pre-trained | FAD database | ASVspoof database
---|---|---|---
clean test | noisy test | 19LA | 21LA
SEEN | unSEEN | SEEN | unSEEN | SEEN | unSEEN | SEEN | unSEEN
LFCC-GMM[11] | - | 6.47 | 31.90 | 29.79 | 30.31 | - | - | - | -
ResNet34 | $\times$ | 0.13(0.13) | 25.98(25.91) | 16.7(16.68) | 37.19(37.09) | 2.36(2.27) | 1.86(1.77) | 16.31(16.06) | 12.97(12.93)
W2V-AASIST[20]$\dagger$ | $\surd$ | 0.08(0.06) | 25.63(25.32) | 2.15(2.03) | 26.79(26.08) | 0.21(0.17) | 0.60(0.42) | 3.32(1.75) | 7.97(5.41)
Ours | $\times$ | 0.30(0.21) | 28.63(26.11) | 4.87(4.11) | 26.88(26.50) | 7.14(6.66) | 7.22(6.81) | 24.69(24.45) | 13.51(12.69)
ASR | 0.05(0.04) | 27.32(25.74) | 3.46(3.11) | 27.69(26.39) | 0.46(0.44) | 0.96(0.94) | 2.2(1.74) | 4.29(4.03)
ASV | 0.14(0.10) | 25.62(25.01) | 3.27(3.04) | 27.25(26.44) | 1.04(0.93) | 1.6(1.53) | 5.57(5.31) | 6.53(5.99)

The system performance is reported using Equal Error Rate (EER). The test data
of a certain spoofing algorithm consists of the spoofing speech of that type
and all the genuine speech from the evaluation set. To assess the robustness
of different models against specific spoofing algorithms, we propose the ET as
a metric. The ET metric quantifies the extent of misjudgment for each specific
spoofing algorithm by a given model. Its value is derived by regularizing the
EER of the model across all spoofing algorithms. Mathematically, it can be
expressed as follows:

$\displaystyle ET=\frac{{M_{i}-\min(M_{i})}}{{\max(M_{i})-\min(M_{i})}}$ (5)

where $M_{i}$ denotes the EER of model $M$ for a specific algorithm $i$. Here,
$i$ belongs to $A07$ to $A19$, which encompasses all spoofing algorithms exist
in the ASVspoof evaluation set. An ET value of 1 indicates that the algorithm
is most likely to be misclassified by model $M$, while a value of 0 indicates
the lowest likelihood of misclassification. It's important to note that the ET
metric does not indicate probability.

When evaluating the subset of the ASVspoof database, it is divided into seen
and unseen portions based on the spoofing algorithm described in [9]. The test
data for the seen category includes A16, A19 spoofing speech, and all genuine
speech. Conversely, the unseen category consists of A10, A11, A12, A13, A14,
A15, A18, and all genuine speech. A07, A08, A09, and A17 are considered as
partial-seen, as part of their algorithms were used during the generation of
the training data.

##  4 Experimental results and analysis

###  4.1 Comparison of CM systems trained with different database

In addition to the official LFCC-GMM baseline, we have also reproduced two
networks as control group models on all databases: a non-pretrained commonly
used CM models, FBANK-ResNet34 [34]; and an SSL pre-trained CM model,
W2V-AASIST [20] 333https://github.com/TakHemlata/SSL_Anti-spoofing. By
comparing the results in Table 2, it can be observed that the pre-trained
model yields a performance improvement over the baseline in all databases.
Furthermore, the proposed Conformer models, pretrained by ASR, significantly
enhance the performance by a large margin on the FAD clean set and the 21LA
dataset. Moreover, all four CM systems utilizing larger models (including the
non-pretrained Conformer model) demonstrate improvements in robustness to
noise when compared to the LFCC-GMM and ResNet34 models. Among them, the
W2V-AASIST model has the best robustness, which indicates that large size
model trained on vast amount of data plays an important role in improving
noise robustness. The fact that the Conformer pre-trained model using only
1/12 of W2V-AASIST's parameters, achieves comparable results, illustrates the
effectiveness of the proposed transfer learning-based Conformer method. We
believe that the small-scale Conformer model can mitigate overfitting and has
great potential in the CM task when model distillation is used.

Table 3: The breakdonw EERs (%) of the CM system on 19LA evaluation set with
different spoofing algorithm.

System | Feature | SEEN | partial SEEN | unSEEN | Pooled
---|---|---|---|---|---
A16 | A19 | A07 | A08 | A09 | A17 | A10 | A11 | A12 | A13 | A14 | A15 | A18
LFCC-GMM [11] | LFCC | 6.31 | 13.94 | 12.86 | 0.37 | 0 | 7.71 | 18.97 | 0.12 | 4.92 | 9.57 | 1.22 | 2.22 | 3.58 | 8.09
Res2Net-C [36] | CQT+phase | 0.34 | 0.33 | 0.22 | 2.67 | 0.02 | 1.75 | 0.51 | 0.33 | 0.18 | 0.06 | 0.22 | 0.22 | 1.77 | 0.94
RawGAT-ST [37] | Waveform | 0.67 | 0.62 | 1.19 | 0.33 | 0.03 | 1.44 | 1.54 | 0.41 | 1.54 | 0.14 | 0.14 | 1.03 | 3.22 | 1.19
AASIST [21] | Waveform | 0.72 | 0.62 | 0.80 | 0.44 | 0 | 1.52 | 1.06 | 0.31 | 0.91 | 0.10 | 0.14 | 0.65 | 3.40 | 0.83
W2V+LGF [19] | Waveform | 0.11 | 0.17 | 0.12 | 0.14 | 0.07 | 0.05 | 3.58 | 3.06 | 0.12 | 0.02 | 0.18 | 0.97 | 0.23 | 1.28
W2V+AASIST | Waveform | 0.02 | 0.30 | 0.04 | 0.02 | 0 | 0.20 | 0.68 | 0.20 | 0.30 | 0 | 0.06 | 0.23 | 0.42 | 0.37
ResNet34 | FBANK | 0.87 | 2.97 | 0.22 | 0.94 | 0.02 | 4.70 | 0.55 | 0.12 | 0.49 | 0.33 | 0.79 | 1.44 | 4.16 | 2.13
Ours (w/o P.) | FBANK | 0.53 | 9.34 | 0.19 | 1.12 | 0.11 | 8.40 | 0.49 | 0.30 | 0.12 | 0.41 | 0.29 | 0.12 | 18.11 | 6.06
Ours (ASR P.) | FBANK | 0.16 | 0.59 | 0.10 | 0.45 | 0.06 | 0.72 | 0.41 | 0.51 | 0.06 | 0 | 0.20 | 0.14 | 2.04 | 0.72
Ours (ASV P.) | FBANK | 0.22 | 1.30 | 0.06 | 0.55 | 0.04 | 1.52 | 0.29 | 0.30 | 0.10 | 0.02 | 0.3 | 0.18 | 3.86 | 1.31
Ours (ASV P.) + W2V-AASIST | - | 0.06 | 0.30 | 0.02 | 0.12 | 0.02 | 0.26 | 0.26 | 0.20 | 0.06 | 0 | 0.08 | 0.08 | 0.88 | 0.31 $\downarrow$
Ours (ASV P.) + Ours (w/o P.) | - | 0.20 | 1.32 | 0.06 | 0.53 | 0.04 | 1.52 | 0.30 | 0.23 | 0.10 | 0.08 | 0.16 | 0.08 | 4.62 | 1.51 $\uparrow$
Spoofing system | Features | System description
A08: NN-based TTS | MCC, F0 | Vocoder is a neural-source-filter waveform model.
A10: NN-based TTS | FBANK | Synthesized audio with speaker information added by WaveRNN vocoder.
A17: NN-based VC | MCC, F0 | Waveform generation is based on a direct waveform modification method.
A18: Non-parallel VC | i-vector, MFCC, F0 | Learning a subspace in the i-vector space that best discriminates speakers.
A19: Transfer-function-based VC | LPCC/MFCC, LPC | Conversion is conducted only on active speech frames.

(a) All the CM systems listed in Table 3.

(b) CM system with raw waveform as input.

(c) CM system with FBANK as input.

Figure 2: The ET curve of CM systems in Table 3.

###  4.2 Analysis of models robustness for certain spoofing algorithm

We refer to the data generated by the spoofing algorithms that have used in
the training set as SEEN, otherwise it is called unSEEN. In Table 2 it is
evident that the data obtained from the unSEEN spoofing methods in the FAD
database are very difficult to distinguish for any model. However, we can
observe a few contrasting conclusions from the ASVspoof database. In
particular, the test results of ResNet34 on the 19LA and 21LA datasets, as
well as the results of Conformer without pre-training on the 21LA dataset,
indicate that the EER of the unSEEN data is lower than that of the SEEN data.
To further investigate the reasons behind this, we performed a re-scoring of
the model on 19LA dataset, breaking down the scores against spoofing
algorithm, and summarized existing breakdown scored system [11, 36, 37, 21,
19] for comparison. From Table 3, we can find out that the performance of CM
systems is not directly dependent on whether they have encountered a spoofing
algorithm before evaluation. For instance, although A19 belongs to SEEN
category, the EER obtained by FBANK is only better than that of the LFCC
baseline. In the unSEEN category, A11 and A13 are relatively easy to
distinguish for each CM system.

The resulting ET metric for all CM systems were calculated, enabling us to
determine which algorithms are more error-prone for different models. The
corresponding results are presented in Figure 2, For a specific system, higher
values indicate a greater error proneness towards that spoofing algorithm.
Since this metric does not represent a probability, it is not feasible to
compare the ET values across CM systems. However, it enables us to observe
whether different models exhibit the same trend of making mistakes for a
certain spoofing algorithm. From Figure 2(b), we observed that the CM systems
using raw waveform as input features are easier to misjudge the A10 and A18
spoofing algorithms in unSEEN category. Conversely, from Figure 2(c), systems
employing FBANK as input features are more likely to misclassify A19, A08,
A17, and A18. It suggest that the FBANK feature-based system exhibits little
significant correlation between its error-prone spoofing algorithms and
whether or not they were encountered in the training set. In Table 3, we
provide a concise summary of five spoofing algorithms that shown poor
performance for most systems, more systems' detail can be found in the paper
[9]. Based on these descriptions, it can be concluded that the CM system faces
difficulties in discriminating data generated by the neural-network-based (NN-
based) TTS system when raw waveform features are used as inputs. Furthermore,
the CM system is more prone to making mistakes when dealing with VC system-
generated data, particularly when FBANK is used as the feature. By referring
to Table 3, we observe that all systems exhibit significantly high EERs on
A18, indicating that the text-independent ASV-based VC system may alter the
speaker information to a large extent without leaving noticeable traces of
synthesis during the creation of spoofing data.

When the ET values of two models are similar, the two models have similar
tendency to make mistakes, and model fusion may lead to counterproductive
results, e.g., ASV-pretrain+w/o pretrain with pooled EER $1.51\uparrow$;
whereas, when the ET curves of the two models are more different, the
performance enhancement of score fusion is more significant, e.g., ASV-
pretrain+W2V-AASIST with pooled EER $0.31\downarrow$.

###  4.3 Comparison with state-of-the-art systems

Table 4 presents a comparison of the proposed ASR/ASV pretrained Conformer
model to the performance of several recently proposed single model [21, 37,
38, 23, 20, 19] on the 19LA [9] dataset. The proposed ASR pretrained Conformer
model, despite having only 13M parameters, outperforms two Wav2Vec 2.0 front-
end models that have over 300M parameters. Moreover, when compared to the
best-performing single-system smaller model, AASIST, the Conformer model
exhibits faster training and inference speeds due to its lack of a complex
graph neural network structure.

Table 4: Comparison with recently proposed state-of-the-art systems, reported
using pooled EER (%) on 19LA evaluation set. Systems are displayed in an
ascending order using the number model parameters. The $\dagger$ model is
implemented without any data augmentation.

System | # Param | Architecture | EER
---|---|---|---
Jung et al. [21] | 297K | AASIST | 0.83
Tak et al. [37] | 437K | RawGAT-ST | 1.06
Zhang et al. [38] | 1,100K | SENet | 1.14
Ours | 13M | ASR pretrained Conformer | 0.72
Ours | 13M | ASV pretrained Conformer | 1.31
Lee et al. [23] | 300+M | W2V(XLSR-53)+ASP | 0.31
Tak et al. [20] $\dagger$ | 300+M | W2V+AASIST | 0.37
Wang et al. [19] | 300+M | W2V(Large2)+LLGF | 0.86
Wang et al. [19] | 300+M | W2V(XLSR-53)+LGF | 1.28

###  4.4 Comparison with other Conformer-based CM systems

Table 5 presents a comparison of the proposed Conformer model and the
performance of other Conformer-based CM systems on 19LA dataset. Rosello et
al. [39] developed CM systems by utilizing classification tokens as output
features and linking FC layers or decoders in the backend. However, as
indicated in Table 5, training of the Conformer model directly on small-scale
anti-spoofing data is vulnerable to overfitting, resulting in degradation of
the generalization performance. Pre-training, on the other hand, can expedite
model fitting and enhance model robustness.

Table 5: Comparison with other Conformer-based CM systems, reported using
pooled EER (%) on 19LA evaluation set.

System | Architecture | EER
---|---|---
Rosello [39] | Conformer + Decoder2 | 7.51
Ours | W/O pretraining | 6.06
ASR pretrained Conformer | 0.72
ASV pretrained Conformer | 1.31

##  5 Conclusion

In this paper, we proposed a CM system based on transfer learning with ASR or
ASV pre-trained MAF-Conformer constructs. We validated the effectiveness of
the proposed method on two different language's anti-spoofing databases, FAD
and ASVspoof. Our results demonstrate that the pretrained model converges
faster and performs better compared to directly training a Conformer model on
anti-spoofing database. Furthermore, when compared to LFCC and ResNet34
models, the ASR pre-trained Conformer model consistently achieves
significantly better results on each database. In addition, we evaluated the
robustness of different models against various spoofing algorithms on the
ASVspoof 2019 LA evaluation set. Our findings clarify that the performance of
neural network-based CM systems is not solely correlated with whether or not
they have seen a spoofing algorithm in training. We propose ET metrics for
measuring the robustness of models to certrain spoofing algorithms. These
metrics may be useful for model fusion and feature selection for spoofing
algorithm traceability tasks. In our future work, we will explore the fusion
of ASV and ASR pre-trained Conformer models along three dimensions: embedding,
score, and model parameters.

## References

  * [1] Zhifu Gao, Yan Song, Ian McLoughlin, Pengcheng Li, Yiheng Jiang, and Li-Rong Dai,  ``Improving Aggregation and Loss Function for Better Embedding Learning in End-to-End Speaker Verification System,''  in Proc. Interspeech, 2019, pp. 361–365.
  * [2] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  ``ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,''  in Proc. Interspeech, 2020, pp. 3830–3834.
  * [3] Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng,  ``Large-scale self-supervised speech representation learning for automatic speaker verification,''  in Proc. ICASSP, 2022, pp. 6147–6151.
  * [4] Aakshi Mittal and Mohit Dua,  ``Automatic speaker verification systems and spoof detection techniques: review and analysis,''  International Journal of Speech Technology, pp. 1–30, 2022.
  * [5] Yoshinori Shiga, Jinfu Ni, Kentaro Tachibana, and Takuma Okamoto,  ``Text-to-speech synthesis,''  Speech-to-Speech Translation, pp. 39–52, 2020.
  * [6] Rohan Kumar Das, Xiaohai Tian, Tomi Kinnunen, and Haizhou Li,  ``The attacker's perspective on automatic speaker verification: An overview,''  arXiv preprint arXiv:2004.08849, 2020.
  * [7] Nicholas Evans, Tomi Kinnunen, Junichi Yamagishi, Zhizheng Wu, Federico Alegre, and Phillip De Leon,  ``Speaker recognition anti-spoofing,''  Handbook of Biometric Anti-Spoofing: Trusted Biometrics under Spoofing Attacks, pp. 125–146, 2014.
  * [8] Choon Beng Tan, Mohd Hanafi Ahmad Hijazi, Norazlina Khamis, Puteri Nor Ellyza binti Nohuddin, Zuraini Zainol, Frans Coenen, and Abdullah Gani,  ``A survey on presentation attack detection for automatic speaker verification systems: State-of-the-art, taxonomy, issues and future direction,''  Multimedia Tools and Applications, vol. 80, no. 21-23, pp. 32725–32762, 2021.
  * [9] Massimiliano Todisco, Xin Wang, Ville Vestman, Md. Sahidullah, Héctor Delgado, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Tomi H. Kinnunen, and Kong Aik Lee,  ``ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection,''  in Proc. Interspeech, 2019, pp. 1008–1012.
  * [10] Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, and Héctor Delgado,  ``ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,''  in Proc. ASVspoof2021 Workshop, 2021, pp. 47–54.
  * [11] Haoxin Ma, Jiangyan Yi, Chenglong Wang, Xinrui Yan, Jianhua Tao, Tao Wang, Shiming Wang, Le Xu, and Ruibo Fu,  ``Fad: A chinese dataset for fake audio detection,''  arXiv preprint arXiv:2207.12308, 2022.
  * [12] Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu,  ``Exploring wav2vec 2.0 on speaker verification and language identification,''  arXiv preprint arXiv:2012.06185, 2020.
  * [13] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  ``wav2vec: Unsupervised pre-training for speech recognition,''  arXiv preprint arXiv:1904.05862, 2019.
  * [14] Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  ``Hubert: How much can a bad teacher benefit asr pre-training?,''  in Proc. ICASSP, 2021, pp. 6533–6537.
  * [15] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  ``wav2vec 2.0: A framework for self-supervised learning of speech representations,''  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [16] Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli,  ``Self-training and pre-training are complementary for speech recognition,''  in Proc., 2021, pp. 3030–3034.
  * [17] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  ``Wavlm: Large-scale self-supervised pre-training for full stack speech processing,''  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [18] Shamane Siriwardhana, Tharindu Kaluarachchi, Mark Billinghurst, and Suranga Nanayakkara,  ``Multimodal emotion recognition with transformer-based self supervised feature fusion,''  IEEE Access, vol. 8, pp. 176274–176285, 2020.
  * [19] Xin Wang and Junichi Yamagishi,  ``Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures,''  in Proc. Odyssey, 2022, pp. 100–106.
  * [20] Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee weon Jung, Junichi Yamagishi, and Nicholas Evans,  ``Automatic Speaker Verification Spoofing and Deepfake Detection Using Wav2vec 2.0 and Data Augmentation,''  in Proc. Odyssey, 2022, pp. 112–119.
  * [21] Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans,  ``Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks,''  in Proc. ICASSP, 2022, pp. 6367–6371.
  * [22] Hemlata Tak, Madhu Kamble, Jose Patino, Massimiliano Todisco, and Nicholas Evans,  ``Rawboost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,''  in Proc. ICASSP, 2022, pp. 6382–6386.
  * [23] Jin Woo Lee, Eungbeom Kim, Junghyun Koo, and Kyogu Lee,  ``Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification,''  in Proc. Interspeech, 2022, pp. 2898–2902.
  * [24] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang,  ``Conformer: Convolution-augmented Transformer for Speech Recognition,''  in Proc. Interspeech, 2020, pp. 5036–5040.
  * [25] Yang Zhang, Zhiqiang Lv, Haibin Wu, Shanshan Zhang, Pengfei Hu, Zhiyong Wu, Hung yi Lee, and Helen Meng,  ``MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification,''  in Proc. Interspeech, 2022, pp. 306–310.
  * [26] D Cai, W Wang, M Li, R Xia, and C Huang,  ``Pretraining conformer with asr for speaker verification,''  in Proc. ICASSP, 2023, pp. 1–5.
  * [27] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov,  ``Transformer-xl: Attentive language models beyond a fixed-length context,''  arXiv preprint arXiv:1901.02860, 2019.
  * [28] Weicheng Cai, Jinkun Chen, Jun Zhang, and Ming Li,  ``On-the-fly data loader and utterance-level aggregation for speaker and language recognition,''  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1038–1051, 2020.
  * [29] David Snyder, Guoguo Chen, and Daniel Povey,  ``MUSAN: A Music, Speech, and Noise Corpus,'' 2015,  arXiv:1510.08484v1.
  * [30] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur,  ``A study on data augmentation of reverberant speech for robust speech recognition,''  in Proc. ICASSP, 2017, pp. 5220–5224.
  * [31] Zhizheng Wu, Tomi Kinnunen, Nicholas Evans, Junichi Yamagishi, Cemal Hanilçi, Md. Sahidullah, and Aleksandr Sizov,  ``ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,''  in Proc. Interspeech, 2015, pp. 2037–2041.
  * [32] Tomi Kinnunen, Md. Sahidullah, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, Junichi Yamagishi, and Kong Aik Lee,  ``The ASVspoof 2017 Challenge: Assessing the Limits of Replay Spoofing Attack Detection,''  in Proc. Interspeech, 2017, pp. 2–6.
  * [33] Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al.,  ``Nemo: a toolkit for building ai applications using neural modules,''  arXiv preprint arXiv:1909.09577, 2019.
  * [34] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman,  ``Voxceleb: Large-scale speaker verification in the wild,''  Computer Speech & Language, vol. 60, pp. 101027, 2020.
  * [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  ``Deep residual learning for image recognition,''  in Proc. CVPR, 2016, pp. 770–778.
  * [36] Juntae Kim and Sung Min Ban,  ``Phase-aware spoof speech detection based on res2net with phase network,''  in Proc. ICASSP, 2023, pp. 1–5.
  * [37] Hemlata Tak, Jee weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, and Nicholas Evans,  ``End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,''  in Proc. ASVspoof 2021 Workshop, 2021, pp. 1–8.
  * [38] Yuxiang Zhang, Wenchao Wang, and Pengyuan Zhang,  ``The Effect of Silence and Dual-Band Fusion in Anti-Spoofing System,''  in Proc. Interspeech, 2021, pp. 4279–4283.
  * [39] Eros Rosello, Alejandro Gomez-Alanis, Manuel Chica, Angel M. Gomez, Jose A. Gonzalez, and Antonio M. Peinado,  ``On the application of conformers to logical access voice spoofing attack detection ,''  in Proc. IberSPEECH, 2022, pp. 181–185.


# Safety-Aware Reinforcement Learning for Electric Vehicle Charging Station
Management in Distribution Network ††thanks: *Corresponding author: Hao Wang
(hao.wang2@monash.edu). ††thanks: This work was supported in part by the
Australian Research Council (ARC) Discovery Early Career Researcher Award
(DECRA) under Grant DE230100046.

Jiarong Fan  Department of Data Science and AI
Faculty of IT, Monash University
Melbourne, Australia
<EMAIL_ADDRESS>Ariel Liebman  Department of Data Science and AI
Faculty of IT, Monash University
Melbourne, Australia
<EMAIL_ADDRESS>Hao Wang  Department of Data Science and AI
Faculty of IT, Monash University
Melbourne, Australia
<EMAIL_ADDRESS>
###### Abstract

The increasing integration of electric vehicles (EVs) into the grid can pose a
significant risk to the distribution system operation in the absence of
coordination. In response to the need for effective coordination of EVs within
the distribution network, this paper presents a safety-aware reinforcement
learning (RL) algorithm designed to manage EV charging stations while ensuring
the satisfaction of system constraints. Unlike existing methods, our proposed
algorithm does not rely on explicit penalties for constraint violations,
eliminating the need for penalty coefficient tuning. Furthermore, managing EV
charging stations is further complicated by multiple uncertainties, notably
the variability in solar energy generation and energy prices. To address this
challenge, we develop an off-policy RL algorithm to efficiently utilize data
to learn patterns in such uncertain environments. Our algorithm also
incorporates a maximum entropy framework to enhance the RL algorithm’s
exploratory process, preventing convergence to local optimal solutions.
Simulation results demonstrate that our algorithm outperforms traditional RL
algorithms in managing EV charging in the distribution network.

###### Index Terms:

Electric vehicle, distribution network, vehicle-to-grid, energy management,
safe reinforcement learning.

##  I Introduction

As a clean mode of transportation, electric vehicles (EVs) will play an
important role in mitigating climate change. Nevertheless, the widespread
adoption of EVs brings substantial challenges to the distribution grid [1].
Specifically, uncoordinated charging of EVs can lead to bus voltage drops,
thereby impacting the grid reliability. Consequently, the deployment of an
intelligent EV charging station management algorithm becomes essential within
the distribution network. However, the coordination algorithm faces the
challenge of handling multiple uncertainties from renewable energy generation
and energy prices. This task demands an effective EV coordination algorithm
optimize energy costs while satisfying the voltage constraints and dealing
with system uncertainties.

A large body of prior research has focused on developing EV charging
management algorithms. By the methodologies, existing studies can be primarily
classified into model-based approaches [2, 3, 4] and model-free approaches [5,
6, 7, 8, 9, 10, 11]. The model-based approach relies on the accurate modeling
of the system and uncertainties. For example, Zheng et al. [2] addressed the
uncertainty of EV charging via model predictive control. However, this
approach is dependent on the prediction accuracy and could suffer from
significant performance degradation with less effective predictions. The non-
convex and non-linear characteristics of network models makes the distribution
network modeling more challenging. The study in [4] relaxed the distribution
network model via the second-order cone relaxation technique, but some
constraints or dynamic behaviors of distribution networks might not be
accurately captured. This approach may oversimplify the model and potentially
overlook critical aspects of the network behavior. Model-free strategies,
particularly those employing reinforcement learning (RL), have shown
significant promise in managing decision-making processes under uncertainties
within complex systems. This advancement offers a potential solution for
addressing the safe EV charging coordination problem in distribution networks
under various uncertainties. For instance, Cao et al. [7] developed a
customized actor-critic (A2C) algorithm to manage EV charging considering
uncertain charging behaviors. In [5], the author used deep RL to learn network
features and maintain network integrity by imposing manually designed voltage
violation penalties.

Nevertheless, the inherent randomness associated with learning-based
algorithms, including RL, brings challenges in satisfying system constraints.
To address this problem, [6] introduced a safe RL algorithm to solve a
constrained Markov decision process (CMDP) problem, thereby avoiding explicit
penalties for constraint violations. An advanced work [10] applied the on-
policy safe RL in the distribution network, but the on-policy RL requires a
large volume of training data and suffers from low data efficiency.
Additionally, [9] introduced an off-policy safe RL algorithm aimed at reducing
the number of voltage violations. However, focusing solely on the frequency of
voltage violations may lead to scenarios where fewer but more severe voltage
violations occur.

This paper is motivated to bridge the aforementioned research gaps by
developing an off-policy safe RL algorithm for coordinating EV charging, which
explicitly considers the trade-off between voltage violation number and
amount. The studied charging station is integrated with solar energy and can
purchase or sell energy through interactions with the grid. The algorithm aims
to minimize the total cost while satisfying system constraints under
uncertainties. The main contributions of this paper are as follows.

  * •

EV charging management via Safe RL: In contrast to the conventional method of
imposing penalties for constraint violations, our method employs a more
systematic approach through the framework of CMDP and using the Soft Actor-
Critic-Lagrangian (SACL) algorithm. Our simulation results demonstrate that
the Lagrange-augmented safe RL algorithm outperforms conventional RL
algorithms in achieving constraint satisfaction.

  * •

Stochastic policy RL with maximum entropy framework: Our proposed SACL is an
RL algorithm that integrates a stochastic policy within a maximum entropy
framework, distinguished by its enhanced exploration capabilities. The
simulation results demonstrate that our algorithm achieves faster convergence
and higher rewards compared to deterministic policy algorithms.

  * •

Trade-off between voltage violation number and amount: Unlike previous studies
only considering the number of voltage violations. Our proposed method
considers the explicit trade-off between the number of voltage violation and
the magnitude.

##  II Problem Formulation

Our study encompasses multiple EV charging stations, with each station
providing services to a set of EVs indexed by $\mathcal{I}{=}{1,...,I}$, each
linked to its respective charger within a defined operational timeframe
$\mathcal{T}$. Every charging station and its solar energy system are
connected to a specific bus in the distribution network. The objective of the
system is to minimize the overall energy costs of all charging stations while
satisfying network constraints.

###  II-1 EV Charging Model

We assume that EV chargers support bidirectional energy flow. The decision
variables for each EV are divided into charging power $p_{i,t}^{\text{ch}}$
and discharging power $p_{i,t}^{\text{disch}}$ at time $t$. The constraints
associated with these decision variables are as follows

$\displaystyle 0\leq p_{i,t}^{\text{ch}}\leq(1-z_{i,t})\bar{P}^{\text{ch}},$
(1) $\displaystyle 0\leq p_{i,t}^{\text{disch}}\leq
z_{i,t}\bar{P}^{\text{disch}},$ (2) $\displaystyle z_{i,t}\in\\{0,1\\},$ (3)

where $\bar{P}^{\text{ch}}$ and $\bar{P}^{\text{disch}}$ denote the maximum
charging and discharging power for EV $i$, respectively. We introduce a binary
variable $z_{i,t}$ to prevent charging and discharging actions at the same
time for all EVs $i\in\mathcal{N}$ in all times $t\in\mathcal{T}$.

EV charging and discharging will lead to the change in the energy level of EV
batteries. The charging station monitors the dynamics of the energy level
$E_{i,t}$ of EV $i$ with the associated constraints expressed as follows

$\displaystyle E_{i,t}=E_{i,t-1}+p_{i,t}^{\text{ch}}\eta^{\text{ch}}_{i}\Delta
t-\frac{p_{i,t}^{\text{disch}}\Delta t}{\eta^{\text{disch}}_{i}},$ (4)
$\displaystyle A_{i,t}E_{i}^{\text{min}}\leq E_{i,t}\leq
A_{i,t}E_{i}^{\text{max}},$ (5)

where $\eta^{\text{ch}}_{i}$ and $\eta^{\text{disch}}_{i}$ denote the charging
and discharging efficiencies for EV $i$, respectively. From the charging
station’s perspective, the observed EV battery’s dynamics can be written in
(4). Equation (5) specifies the minimum and maximum energy levels
$E_{i}^{\text{min}}$ and $E_{i}^{\text{max}}$ for the $i$-th EV’s battery with
$A_{i,t}$ indicating the connection status of the $i$-th EV at $t$. When
$A_{i,t}=0$, EV $i$ is disconnected from the charger and thus the observed
$E_{i,t}=0$. To satisfy constraint (5), both $p_{i,t}^{\text{ch}}$ and
$p_{i,t}^{\text{disch}}$ must be set to 0 according to Equation (4), when
$A_{i,t}=0$.

When assessing the economic viability of EV charging and discharging, it is
vital to consider the additional costs incurred by battery degradation.
According to the energy throughput equivalent method [12], the $i$-th EV’s
cycle aging denoted $AGE_{i,t}^{\text{cyc}}$, can be modeled below

$\displaystyle AGE_{i,t}=0.5\cdot\frac{\lvert
p_{i,t}^{\text{ch}}\eta^{\text{ch}}_{i}\Delta
t-\frac{p_{i,t}^{\text{disch}}\Delta
t}{\eta^{\text{disch}}_{i}}\rvert}{E_{i}^{\text{cap}}L_{i}^{\text{cyc}}},$ (6)

in which $E_{i}^{\text{cap}}$ denotes the capacity of the $i$-th EV’s battery,
while the value of 0.5 represents the average half-cycle of the battery’s
life. The fraction of battery aging that occurs within a single time step
relative to the overall lifecycle of the battery is expressed as
$L_{i}^{\text{cyc}}$.

###  II-2 Energy Balance and Network Model

In our analysis, the distribution network is modeled as a graph $\mathcal{G}$
= {$\mathcal{N},\mathcal{E}$}, comprising a set of nodes
$\mathcal{N}_{0}=\\{0,1,...,N\\}$ and edges $\mathcal{E}$, with node $0$
designated as the substation. The subset of nodes
$\mathcal{N}=\mathcal{N}_{0}\backslash\\{0\\}$ represents all nodes excluding
the substation node $0$. We denote $n$ is the parent node of $m$. Each node
$n\in\mathcal{N}$ is characterized by active power injection $p_{n}$, reactive
power injection $q_{n}$, and voltage magnitude $v_{n}$. We adopt the
linearized distribution power flow model, specifically the simplified distflow
model, which is a linear approximation of the nonlinear branch flow model for
radial distribution networks. This approximation is obtained by disregarding
the terms for quadratic power loss, and the simplified distflow model is well-
established in the area of voltage control. In this model, the variables $p$,
$q$, and $v$ satisfy the following equations

$\displaystyle\begin{split}&P_{nm,t}-\sum_{g\in G(n)}p_{ng,t}=\sum_{c\in
S(n)}p_{c,t}^{\text{sell},n}\\\ &-\sum_{c\in
S(n)}p_{c,t}^{\text{buy},n}+p_{n,t}^{\text{V2G}}-p_{n,t}^{\text{G2V}}-p_{n,t}^{\text{base}},\\\
\end{split}$ (7) $\displaystyle\begin{split}&Q_{nm,t}-\sum_{g\in
G(n)}q_{ng,t}=\sum_{c\in S(n)}q_{c,t}^{\text{sell},n}\\\ &-\sum_{c\in
S(n)}q_{c,t}^{\text{buy},n}+q_{n,t}^{\text{V2G}}-q_{n,t}^{\text{G2V}}-q_{n,t}^{\text{base}},\\\
\end{split}$ (8) $\displaystyle
v^{2}_{m,t}=v^{2}_{n,t}-2(r_{nm,t}p_{nm,t}+x_{nm,t}q_{nm,t}),$ (9)
$\displaystyle\underline{V}\leq v_{n,t}\leq\overline{V},$ (10)

where $G(n)$ represents the set of child buses connected to node $n$ , while
$S(n)$ denotes the set of charging stations excluding node $n$. The
active/reactive power injection of each node includes energy trading power
$p_{c,t}^{\text{sell},n}$/$q_{c,t}^{\text{sell},n}$ and
$p_{c,t}^{\text{buy},n}$/$q_{c,t}^{\text{buy},n}$, V2G power
$p_{n,t}^{\text{V2G}}$/$q_{n,t}^{\text{V2G}}$, grid purchased power
$p_{n,t}^{\text{G2V}}$/$q_{n,t}^{\text{G2V}}$, and base load
$p_{n,t}^{\text{base}}$/$q_{n,t}^{\text{base}}$.

Additionally, the voltage level at each bus is assumed to be close to the
reference voltage $v_{0}$, such that (9) is rewritten as the following linear
form

$\displaystyle
v_{m,t}=v_{n,t}-\frac{(r_{nm,t}p_{nm,t}+x_{nm,t}q_{nm,t})}{v_{0}}.$ (11)

###  II-3 Charging Completion

Each EV must meet its charging requirements before leaving the charging
station. The departure time for the $i$-th EV is denoted as $t=dep_{i}$.
Failure to meet the charging requirement will incur a penalty, denoted by
$u_{i}$. For simplicity, [8] suggests a linear form for $u_{i}$ below

$\displaystyle
u_{i}=\sigma(E_{i}^{\text{dem}}-E_{i,t}\big{|}_{t=dep_{i}})^{+},$ (12)

where $\sigma$ signifies a constant coefficient to balance the costs
associated with charging and completion of charging tasks. The charging
requirement for the $i$-th EV, denoted as $E_{i}^{\text{dem}}$, is
characterized as a time-varying parameter.

###  II-4 Objectives and Constraints

The operational goals of the charging station service encapsulate two primary
objectives: energy cost reduction and satisfaction of charging demands. The
objective function, along with operational constraints, is shown as follows

$\displaystyle\begin{split}&\min~{}~{}\\!\\!\\!\\!\sum_{t\in\mathcal{T}}\Bigl{(}\sum_{i\in\mathcal{I}}(AGE_{i,t}^{\text{cyc}}\kappa^{\text{batt}})\\\
&+\sum_{n\in\mathcal{N}}\bigl{(}(p_{n,t}^{\text{G2V}}\kappa_{t}^{\text{buy}}-(p_{n,t}^{\text{V2G}}+p_{n,t}^{\text{PVG}})\kappa_{t}^{\text{sell}})\Delta
t\bigr{)}\Bigr{)}+\sum_{i\in\mathcal{I}}u_{i}\\\
&\text{s.t.}~{}~{}\eqref{xt_con}-\eqref{eit_con}~{}\text{and}~{}\eqref{act}-\eqref{volt_con},\end{split}$
(13)

where $\kappa^{\text{buy}}_{t}$ and $\kappa^{\text{sell}}_{t}$ are energy
selling price and energy purchasing price at time $t$. Energy trading revenues
and costs offset each other when calculating the overall system cost.

##  III Constrained Markov Decision Process

We approach the safety-aware EV charging/discharging coordination problem by
formulating it as a CMDP and solve it using safe RL techniques. In this
context, the distribution system operator (DSO) acts as an agent managing EV
charging/discharging together with DERs scheduling within the distribution
network, with a crucial focus on maintaining network security. Our CMDP
formulation comprises several key components, including state, action, reward,
and a auxiliary cost function.

State: The state includes necessary information for the calculation of
costs/rewards and constraints. It has two primary segments: EV charging
information, including the remaining charging demand $E^{r}_{i,t}$ and time
$T^{r}_{i,t}$, and the environmental information, including solar power
generation $p_{t}^{\text{PVgen}}$, electricity selling price
$\kappa_{t}^{\text{sell}}$, purchasing price $\kappa_{t}^{\text{buy}}$, and
bus voltage $v_{n}$.

Action: The actions in this RL environment include charging/discharging power
$p_{i,t}$ for the $i$-th EV at time $t$ and the energy trading decisions
$E_{n,t}^{\text{trade}}$ for the $n$-th charging station at time $t$, which
encompass the proportions of the intended trading volumes for each station.
Based on these trading actions, the environment will execute the decisions
across charging stations, with any remaining power to be sold to or purchased
from the grid.

Reward: The RL algorithm aims to generate an optimal policy by maximizing the
model’s reward. The objective is to minimize the total system cost while
ensuring compliance with system constraints. A negative reward, $R_{t}$, is
employed to capture the total energy cost and user satisfaction, aligning with
the objective function in Equation (13). Additionally, an auxiliary cost
function is utilized within the CMDP to handle the system constraints.

Auxiliary Cost Function:  To avoid manual reward design and adjustment, we
tranform the original Markov decision process model to a CMDP model. We
specify the auxiliary cost function $c_{t}$ as follows

$\displaystyle
c_{t,n}=\sum_{n=0}^{N}[\textbf{1}(|v_{n,t}|>\overline{V})+\textbf{1}(|v_{n,t}|>\underline{V})],$
(14) $\displaystyle
c_{t,a}=\sum_{n=0}^{N}[\text{max}(0,|v_{n,t}|-\overline{V})+\text{max}(0,\underline{V}-|v_{n,t}|)],$
(15) $\displaystyle c_{t}=\beta c_{t,n}+(1-\beta)c_{t,a},$ (16)

where $c_{t,n}$ represents the number of voltage violations, while $c_{t,a}$
corresponds to the magnitude of these violations. We use $\textbf{1}(\cdot)$
to represent the indicator function. Its value is $1$ when the internal
condition is met and 0 otherwise. The upper bound and lower bound of the bus
voltage are represented as $\overline{V}$ and $\underline{V}$, respectively.
The parameter $\beta\in[0,1]$ is a trade-off weight between the voltage
violation number and amount.

##  IV Our Proposed Safe RL Method

In this section, we develop a safe RL algorithm to learn a strategy for
managing EV charging stations with the dual objectives of optimizing the
cumulative reward and adhering to system constraints within the distribution
network. Insufficient exploration of the environment may lead to the agent
falling into a local optimum. To avoid this, we employ the Soft Actor-Critic
(SAC) method [9], which incorporates a maximum entropy framework to enhance
the randomness of action selection and improve the agent’s exploration
ability. To address system constraints, we use the Lagrange function that
integrates network constraints into the value function, eliminating the need
for penalty tuning.

The SACL framework, depicted in Fig. 1, is based on the A2C architecture. The
neural network is used to estimate the function of SACL framework via a data-
driven approach. These neural networks share the same architecture with two
hidden layers and a rectified linear activation function. Within this
framework, the actor network estimates the state-value function and determines
the actions for the DSO. Concurrently, the critic network estimates the
action-value function and assesses the actions taken by the actor, which aims
to reduce the temporal difference error. For Lagrange function, an additional
neural network is utilized to process the Lagrange multiplier. The Lagrange
network is integrated into our framework to dynamically adjust the penalty
associated with constraint violations, ensuring efficient learning without
manual tuning of penalty coefficients. This network also features two hidden
layers, using ReLU activation functions for hidden layers to support complex
function approximation while maintaining computational efficiency.

Figure 1: The architecture of the SACL.

Within this framework, the goal extends beyond merely maximizing the expected
return; it also includes maximizing the entropy of the policy, thereby
fostering more exploratory policy behavior. Within the context of maximum-
entropy RL, the regularized state-value function is defined as follows

$\displaystyle
V_{\text{soft}}(\textbf{s})=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{T}\gamma(R_{t}+\alpha
H(\pi(\cdot|\textbf{s}_{t})))|\textbf{s}_{0}=s],$ (17)

where $H(\pi(\cdot|\textbf{s}_{t})))$ represents the entropy of a
probabilistic policy at state $\textbf{s}_{t}$ with $\pi$ serving as the
function that maps a state to actions, and $\gamma$ is the discount factor.
The action-value function is written as

$\displaystyle\begin{split}&Q_{\text{soft}}(\textbf{s},\textbf{a})=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{T}\gamma
R_{t}\\\ &+\alpha\sum_{t=0}^{T}\gamma
H(\pi(\cdot|\textbf{s}_{t})))|\textbf{s}_{0}=s,\textbf{a}_{0}=a].\end{split}$
(18)

To deal with system constraints, we define $J_{c}(\pi)$ as the expected
discounted return of the policy $\pi$ with respect to the auxiliary cost
function: $J_{c}(\pi)=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{T}\gamma c_{t}]$.
For the SACL framework, the optimal policy of CMDP can be obtained by solving

$\displaystyle\max_{\pi}\mathbb{E}_{\textbf{s}\sim\textbf{D}}[V_{\text{soft}}(\textbf{s})],\;s.t\;J_{c}(\pi)<d,$
(19)

where D is the historical data replay buffer, and $d$ is a small tolerance for
the voltage constraint violation. In order to solve (19), the optimization
problem with constraints can be converted to a Lagrange function written as

$\displaystyle\mathcal{L}(\pi,\lambda)=\mathbb{E}_{\textbf{s}\sim\textbf{D}}[V_{\text{soft}}(\textbf{s})]+\lambda(d-J_{c}(\pi)),$
(20)

where $\lambda$ is the Lagrange multiplier to be updated at each iteration.
The updating rules for value functions and $\lambda$ are presented in [9].

##  V PERFORMANCE EVALUATION

We take the perspective of the DSO to train an agent for managing a
distribution network and use the IEEE 33-bus system in the simulations. We use
PandaPower to build the simulation system using its default base load data.
Four charging stations are placed at node 8, node 12, node 22, and node 30.
Each charging station has five EV chargers with a charging power of 22kW and
solar panels with a capacity of 13kWp. EV charging data are obtained from
[13]. EVs purchase energy following Time-of-Use (TOU) pricing, while energy is
sold to the grid at wholesale energy prices [14]. The energy trading price is
set as the midpoint of the TOU energy price and the wholesale market energy
price. Solar energy, which is free to use, is simulated using solar energy
generation data from [15].

To effectively manage charging stations and meet system constraints, we
implement the SACL algorithm, handling system constraints through the
application of the Lagrange function. SACL enhances environmental exploration
and prevents convergence on suboptimal local solutions via its stochastic
policy embedded within a maximum entropy framework. The hyper-parameters of
SACL are determined through the empirical testing to identify configurations
that yield the best performance across various simulation scenarios. For
comparative analysis, we also consider Deep Deterministic Policy Gradient
(DDPG) and SAC algorithms, as DDPG is a representative deterministic policy
algorithm and SAC employs penalties for constraint violations.

Fig. 2 illustrates the convergence of rewards for these algorithms. All three
algorithms demonstrate convergence within 100 episodes. SAC and SACL exhibit
faster convergence and achieve higher rewards compared to DDPG, attributed to
the maximum entropy framework that expedites the exploration of more effective
policies.

Figure 2: The reward convergence for different algorithms.

Fig. 3 shows the net power output at a single charging station under various
uncertainties. It is clear that our proposed algorithm successfully captures
the pattern of solar energy generation, optimizing the use of freely available
solar energy. Furthermore, the algorithm guides the charging station to sell
energy back to the grid when the selling price exceeds the purchasing price.
The results depicted in Fig. 3 demonstrate the effectiveness of the proposed
algorithm.

Figure 3: Net power of one charging station under uncertainties of solar
energy and real-time price.

To regulate bus voltage, we consider two parameters: voltage violation number
(VVN), as defined in (14), and voltage violation amount (VVA), outlined in
(15). Focusing solely on VVN can lead to a few severe voltage violations,
while an exclusive emphasis on VVA could result in numerous minor violations.
Therefore, achieving a balance between VVN and VVA is crucial. Fig. 4
illustrates this trade-off through a frequency histogram. As the trade-off
parameter $\beta$ decreases, the emphasis shifts towards VVA, resulting in an
increased number of violations. Conversely, a higher value of $\beta$
($\beta=1$) reduces VVN but may lead to more severe voltage violations.

Figure 4: The frequency histogram for voltage violation amount.

Table I compares energy cost reduction and voltage satisfaction performance
among different algorithms. With consistent parameter settings, SAC and SACL
outperform DDPG in energy cost reduction and charging demand satisfaction, due
to their superior exploration capabilities. Regarding voltage constraints, the
VVN and VVA of safety-aware SACL are lower than those of SAC, underlining
SACL’s effectiveness in satisfying voltage constraints.

TABLE I: Performance comparisons between SACL, SAC, and DDPG in terms of cost, unfinished demand (UD), voltage violation number (VVN), and voltage violation amount (VVA). Algorithm | Cost ($) | UD (kWh) | VVN | VVA (p.u.)
---|---|---|---|---
DDPG | 1091 | 412.7 | 551 | 1.611
SAC | 933 | 427.5 | 436 | 1.284
SACL | 926 | 411.2 | 294 | 0.915

##  VI Conclusion and Future Work

Our work developed an innovative safety-aware reinforcement learning approach
for managing multiple EV charging stations in distribution networks. Our
proposed algorithm excels in striking a balance between cost optimization and
adherence to system constraints, particularly under uncertainties including
fluctuating energy prices and solar energy generation. Our simulation results
demonstrate that our algorithm outperforms widely used reinforcement learning
methods by leveraging the maximum-entropy framework and the Lagrangian
function. Future research will consider reactive power for enhanced voltage
control and seek to improve the scalability of the algorithm to coordinate
more charging stations in larger distribution networks.

## References

  * [1] K. Clement-Nyns, E. Haesen, and J. Driesen, “The impact of charging plug-in hybrid electric vehicles on a residential distribution grid,” _IEEE Transactions on power systems_ , vol. 25, no. 1, pp. 371–380, 2009.
  * [2] Y. Zheng, Y. Song, D. J. Hill, and K. Meng, “Online distributed mpc-based optimal scheduling for ev charging stations in distribution systems,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 2, pp. 638–649, 2018.
  * [3] O. Beaude, S. Lasaulce, M. Hennebel, and I. Mohand-Kaci, “Reducing the impact of ev charging operations on the distribution network,” _IEEE Transactions on Smart Grid_ , vol. 7, no. 6, pp. 2666–2679, 2016.
  * [4] L. Luo, W. Gu, Z. Wu, and S. Zhou, “Joint planning of distributed generation and electric vehicle charging stations considering real-time charging navigation,” _Applied energy_ , vol. 242, pp. 1274–1284, 2019.
  * [5] J. Duan, D. Shi, R. Diao, H. Li, Z. Wang, B. Zhang, D. Bian, and Z. Yi, “Deep-reinforcement-learning-based autonomous voltage control for power grid operations,” _IEEE Transactions on Power Systems_ , vol. 35, no. 1, pp. 814–817, 2019.
  * [6] H. Li, Z. Wan, and H. He, “Constrained ev charging scheduling based on safe deep reinforcement learning,” _IEEE Transactions on Smart Grid_ , vol. 11, no. 3, pp. 2427–2439, 2019.
  * [7] Y. Cao, H. Wang, D. Li, and G. Zhang, “Smart online charging algorithm for electric vehicles via customized actor–critic learning,” _IEEE Internet of Things Journal_ , vol. 9, no. 1, pp. 684–694, 2021.
  * [8] F. Tuchnitz, N. Ebell, J. Schlund, and M. Pruckner, “Development and evaluation of a smart charging strategy for an electric vehicle fleet based on reinforcement learning,” _Applied Energy_ , vol. 285, p. 116382, 2021.
  * [9] W. Wang, N. Yu, Y. Gao, and J. Shi, “Safe off-policy deep reinforcement learning algorithm for volt-var control in power distribution systems,” _IEEE Transactions on Smart Grid_ , vol. 11, no. 4, pp. 3008–3018, 2019.
  * [10] H. Li and H. He, “Learning to operate distribution networks with safe deep reinforcement learning,” _IEEE Transactions on Smart Grid_ , vol. 13, no. 3, pp. 1860–1872, 2022.
  * [11] K. Park and I. Moon, “Multi-agent deep reinforcement learning approach for ev charging scheduling in a smart grid,” _Applied Energy_ , vol. 328, p. 120111, 2022.
  * [12] Y. Wu, Z. Liu, J. Liu, H. Xiao, R. Liu, and L. Zhang, “Optimal battery capacity of grid-connected pv-battery systems considering battery degradation,” _Renewable Energy_ , vol. 181, pp. 10–23, 2022.
  * [13] Z. J. Lee, T. Li, and S. H. Low, “Acn-data: Analysis and applications of an open ev charging dataset,” in _Proceedings of the Tenth ACM International Conference on Future Energy Systems_ , 2019, pp. 139–149.
  * [14] AEMO, “Nem data dashboard,” 2023, https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/data-dashboard-nem.
  * [15] Elia, “griddata,” 2021, https://www.elia.be/en/grid-data.