$\displaystyle=h(X)-h(Y)-\textnormal{Tr}[(X-Y)^{\dagger}\nabla h(Y)]$
$\displaystyle=\textnormal{Tr}[X\log X]-\textnormal{Tr}[Y\log
Y]-\textnormal{Tr}[(X-Y)^{\dagger}\nabla\textnormal{Tr}[Y\log Y]]$
$\displaystyle=\textnormal{Tr}[X\log X]-\textnormal{Tr}[Y\log
Y]-\textnormal{Tr}[(X-Y)^{\dagger}(\mathbb{1}+\log Y)]$
$\displaystyle=\textnormal{Tr}[X\log X]-\textnormal{Tr}[Y\log
Y]-\textnormal{Tr}[X]+\textnormal{Tr}[Y]-\textnormal{Tr}[X\log
Y]+\textnormal{Tr}[Y\log Y]$ $\displaystyle=\textnormal{Tr}[X(\log X-\log
Y)].$

The regularized best-response,

$\displaystyle\textnormal{Mir}\Pi_{\mathcal{C}}^{h}(Y)={\arg\min}_{C\in\mathcal{C}}\\{\textnormal{Tr}[Y^{\dagger}C]-\textnormal{Tr}[C\log
C]\\}$

has Lagrangian

$\displaystyle\mathcal{L}(Y)=\textnormal{Tr}[Y^{\dagger}C]-\textnormal{Tr}[C\log
C]-\lambda\cdot(\textnormal{Tr}[C]-1),$

with gradients

$\displaystyle\nabla\mathcal{L}=\begin{cases}\nabla_{C^{\top}}\mathcal{L}(Y)&=Y-(\mathbb{1}+\log
C)-\lambda\mathbb{1}=Y-\log C-\alpha\mathbb{1}\\\
\nabla_{\lambda}\mathcal{L}(Y)&=-\textnormal{Tr}[C]+1,\end{cases}$

where $\alpha=\lambda+1$. Setting the gradients to zero results in the system
of equations

$\displaystyle\begin{cases}\log C&=Y-\alpha\mathbb{1}\\\
\textnormal{Tr}[C]&=1\end{cases}\implies\begin{cases}C&=\exp(Y-\alpha\mathbb{1})=\exp(Y)\cdot
e^{-\alpha}\mathbb{1}\\\ \textnormal{Tr}[C]&=1.\end{cases}$

We can solve for $\alpha$ by plugging the first equation into the second and
using the properties of matrix exponentials as follows.

$\displaystyle\textnormal{Tr}[\exp(Y)\cdot
e^{-\alpha}\mathbb{1}]=e^{-\alpha}\textnormal{Tr}[\exp(Y)]=1\>\>\implies\>\>e^{-\alpha}=\frac{1}{\textnormal{Tr}[\exp(Y)]}.$

Substituting this back into our equation for $C$ gives

$\displaystyle C=\frac{\exp(Y)}{\textnormal{Tr}[\exp(Y)]}.$

Therefore, as desired, the regularized best response is the logit map

$\displaystyle\textnormal{Mir}\Pi_{\mathcal{C}}^{h}(Y)=\frac{\exp(Y)}{\textnormal{Tr}[\exp(Y)]}=\Lambda(Y).$

Finally, the corresponding proximal projection is given by

$\displaystyle\textnormal{Prox}\Pi^{h}_{\mathcal{C}}(X,Y)={\arg\min}_{C\in\mathcal{C}}\left\\{\textnormal{Tr}[Y^{\dagger}(X-C)]+\textnormal{Tr}[C(\log
C-\log X)]\right\\},$ (162)

which has Lagrangian

$\displaystyle\mathcal{L}(Y)=\textnormal{Tr}[Y^{\dagger}(X-C)]+\textnormal{Tr}[C(\log
C-\log X)]-\lambda\cdot(\textnormal{Tr}[C]-1),$

with gradients

$\displaystyle\nabla\mathcal{L}=\begin{cases}\nabla_{C^{\top}}\mathcal{L}(Y)&=-Y+\mathbb{1}+\log
C-\log X-\lambda\mathbb{1}=\log C-Y-\log X+\alpha\mathbb{1}\\\
\nabla_{\lambda}\mathcal{L}(Y)&=-\textnormal{Tr}[C]+1,\end{cases}$

where $\alpha=\lambda+1$. Setting the gradients to zero results in the system
of equations

$\displaystyle\begin{cases}\log C&=Y+\log X-\alpha\mathbb{1}\\\
\textnormal{Tr}[C]&=1\end{cases}\implies\begin{cases}C&=\exp(Y+\log X)\cdot
e^{-\alpha}\mathbb{1}\\\ \textnormal{Tr}[C]&=1\end{cases}$

We can solve for $\alpha$ by plugging the first equation into the second and
using the properties of matrix exponentials as follows.

$\displaystyle\textnormal{Tr}[\exp(Y+\log X)\cdot
e^{-\alpha}\mathbb{1}]=e^{-\alpha}\textnormal{Tr}[\exp(Y+\log
X)]=1\>\>\implies\>\>e^{-\alpha}=\frac{1}{\textnormal{Tr}[\exp(Y+\log X)]}.$

Substituting this back into our equation for $C$ gives

$\displaystyle C=\frac{\exp(Y+\log X)}{\textnormal{Tr}[\exp(Y+\log X)]}.$

Therefore, as desired, the regularized best response is the logit map

$\displaystyle\textnormal{Prox}\Pi^{h}_{\mathcal{C}}(X,Y)=\frac{\exp(Y+\log
X)}{\textnormal{Tr}[\exp(Y+\log X)]}=\Lambda(Y+\log X).$

$\square$

##  Appendix C Properties of Finite-Valued Zero-Sum Quantum Games

In this appendix we will prove several important properties of finite-valued
zero-sum quantum games.

###  C.1 Properties of Spectraplexes

###### Lemma 10

The spectraplex
$\mathcal{X}=\\{X\in\mathcal{H}^{k}_{+}:\textnormal{Tr}(X)=1\\}$ is a convex
set.

#####  Proof.

By the definition of convex set, $\mathcal{X}$ is a convex set iff, for any
$X_{1},X_{2}\in\mathcal{X}$ and $t\in(0,1]$,

$\displaystyle\tilde{X}=(1-t)\cdot X_{1}+t\cdot X_{2}$ (163)

such that $\tilde{X}\in\mathcal{H}^{k}_{+}$ and
$\textnormal{Tr}(\tilde{X})=1$.

We begin by proving that $\tilde{X}$ is positive semi-definite,
$\tilde{X}\in\mathcal{H}^{k}_{+}$, which is true iff
$\forall\ket{v}\in\mathbb{C}^{k}$,

$\displaystyle\bra{v}\tilde{X}\ket{v}\geq 0.$ (164)

Given that $X_{1},X_{2}\in\mathcal{H}^{k}_{+}$,

$\displaystyle\forall\ket{v}\in\mathbb{C}^{k}:\>\>\>\bra{v}X_{1}\ket{v}\geq
0\>\>\text{ and }\>\>\bra{v}X_{2}\ket{v}\geq 0.$ (165)

Therefore, since $t\geq 0$ and $(1-t)\geq 0$,

$\displaystyle\bra{v}\tilde{X}\ket{v}=\bra{v}(1-t)\cdot X_{1}+t\cdot
X_{2}\ket{v}=(1-t)\bra{v}X_{1}\ket{v}+t\bra{v}X_{2}\ket{v}\geq 0.$ (166)

We now prove that $\textnormal{Tr}(\tilde{X})=1$. Since
$\textnormal{Tr}[X_{1}]=\textnormal{Tr}[X_{2}]=1$,

$\displaystyle\textnormal{Tr}(\tilde{X})=\textnormal{Tr}\Big{[}(1-t)\cdot
X_{1}+t\cdot
X_{2}\Big{]}=(1-t)\cdot\textnormal{Tr}[X_{1}]+t\cdot\textnormal{Tr}[X_{2}]=1-t+t=1$
(167)

$\square$

###### Lemma 11

The spectraplex
$\mathcal{X}=\\{X\in\mathcal{H}^{k}_{+}:\textnormal{Tr}(X)=1\\}$ is a compact
set.

#####  Proof.

See page 162 of [75]. $\square$

###  C.2 Properties of $\mathcal{F}$

See 8

#####  Proof.

For this proof, we will leverage the Pauli decompositions of the involved
operators. Note that every matrix $M$ can be decomposed in terms of the Pauli
matrices $\mathcal{P}=\\{I,X,Y,Z\\}$, as

$\displaystyle M=\sum_{P\in\mathcal{P}^{\otimes n}}\widehat{M}(P)P,$

where the tensored Pauli matrices $P$ constitute an orthonormal basis, with
respect to trace inner product, and possess corresponding Pauli coefficients
$\widehat{M}(P)$. Note that for all Paulis, we have $P=P^{\dagger}$. The Pauli
coefficients of an $n$-qubit system can also be explicitly defined as

$\widehat{M}(P)=\frac{1}{2^{n}}\textnormal{Tr}(P^{\dagger}M).$

The payoff observable $U$ can be decomposed as

$\displaystyle
U=\sum_{R\in\mathcal{P}^{\otimes(nm)}}\widehat{U}(R)R=\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes m}}\widehat{U}(P,Q)P\otimes Q,$

where $\widehat{U}(P,Q)=\widehat{U}(P\otimes Q)$. Plugging this into the
payoff gradients defined in Equation 16 and Equation 17, we get that

$\displaystyle\mathcal{F}_{\alpha}(\alpha,\beta)$
$\displaystyle=\nabla_{\alpha^{\top}}\textnormal{Tr}\left(U^{\dagger}(\alpha\otimes\beta)\right)$
$\displaystyle=\nabla_{\alpha^{\top}}\textnormal{Tr}\left(\left(\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes m}}\widehat{U}(P,Q)P\otimes
Q\right)^{\dagger}(\alpha\otimes\beta)\right)$
$\displaystyle=\nabla_{\alpha^{\top}}\textnormal{Tr}\left(\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}P^{\dagger}\alpha\otimes Q^{\dagger}\beta\right)$
$\displaystyle=\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}\cdot\nabla_{\alpha^{\top}}\textnormal{Tr}\left(P^{\dagger}\alpha\otimes
Q^{\dagger}\beta\right)$ $\displaystyle=\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}\cdot\nabla_{\alpha^{\top}}\textnormal{Tr}\left(P^{\dagger}\alpha\right)\textnormal{Tr}\left(Q^{\dagger}\beta\right)$
$\displaystyle=\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes m}}\widehat{U}(P,Q)^{*}\cdot
2^{m}\widehat{\beta}(Q)\>P^{\dagger}$

and, similarly,

$\displaystyle\mathcal{F}_{\beta}(\alpha,\beta)$
$\displaystyle=-\nabla_{\beta^{\top}}\textnormal{Tr}\left(U^{\dagger}(\alpha\otimes\beta)\right)$
$\displaystyle=-\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}\cdot\nabla_{\beta^{\top}}\textnormal{Tr}\left(P^{\dagger}\alpha\right)\textnormal{Tr}\left(Q^{\dagger}\beta\right)$
$\displaystyle=-\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes m}}\widehat{U}(P,Q)^{*}\cdot
2^{n}\widehat{\alpha}(P)\>Q^{\dagger}.$

Thus, plugging these results into Equation 19, the operator $\mathcal{F}$ can
be decomposed as

$\displaystyle\mathcal{F}(\alpha,\beta)$
$\displaystyle=\begin{pmatrix}2^{m}&\sum_{P\in\mathcal{P}^{\otimes
n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}\cdot\widehat{\beta}(Q)\>P^{\dagger}\\\
-2^{n}&\sum_{P\in\mathcal{P}^{\otimes n}}\sum_{Q\in\mathcal{P}^{\otimes
m}}\widehat{U}(P,Q)^{*}\cdot\widehat{\alpha}(P)\>Q^{\dagger}\end{pmatrix}.$
(168)

Now, let us consider the two distinct games states, $X=(x_{1},x_{2})$ and
$Y=(y_{1},y_{2})$, where $x_{1},y_{1}\in\mathcal{A}$ and
$x_{2},y_{2}\in\mathcal{B}$, with difference

$\displaystyle X-Y$ $\displaystyle=\begin{pmatrix}x_{1}\\\
x_{2}\end{pmatrix}-\begin{pmatrix}y_{1}\\\
y_{2}\end{pmatrix}=\begin{pmatrix}x_{1}-y_{1}\\\ x_{2}-y_{2}\end{pmatrix}.$

Following from Equation 168, the difference of the payoff gradients is

$\displaystyle F(X)$
$\displaystyle-F(Y)=F(x_{1},x_{2})-F(y_{1},y_{2})=\begin{pmatrix}2^{m}\sum_{P,Q}\widehat{U}(P,Q)^{*}\cdot\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\>P^{\dagger}\\\
2^{n}\sum_{P,Q}\widehat{U}(P,Q)^{*}\cdot\left(\widehat{y_{1}}(P)-\widehat{x_{1}}(P)\right)\>Q^{\dagger}\end{pmatrix}$

Plugging these into the left-hand side of Equation 73, we get that

Tr $\displaystyle\left[(\mathcal{F}(X)-\mathcal{F}(Y))(X-Y)\right]=$
$\displaystyle=\textnormal{Tr}\left[\begin{pmatrix}2^{m}\sum_{P,Q}\widehat{U}(P,Q)^{*}\cdot\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\>P^{\dagger}\\\
2^{n}\sum_{P,Q}\widehat{U}(P,Q)^{*}\cdot\left(\widehat{y_{1}}(P)-\widehat{x_{1}}(P)\right)\>Q^{\dagger}\end{pmatrix}\begin{pmatrix}x_{1}-y_{1}\\\
x_{2}-y_{2}\end{pmatrix}\right]$
$\displaystyle=\textnormal{Tr}\left[\sum_{P,Q}\widehat{U}(P,Q)^{*}\left(2^{m}\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\>P^{\dagger}(x_{1}-y_{1})+2^{n}\left(\widehat{y_{1}}(P)-\widehat{x_{1}}(P)\right)\>Q^{\dagger}(x_{2}-y_{2})\right)\right]$
$\displaystyle=\sum_{P,Q}\widehat{U}(P,Q)^{*}\left(2^{m}\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\>\textnormal{Tr}\left[P^{\dagger}(x_{1}-y_{1})\right]+2^{n}\left(\widehat{y_{1}}(P)-\widehat{x_{1}}(P)\right)\>\textnormal{Tr}\left[Q^{\dagger}(x_{2}-y_{2})\right]\right)$
$\displaystyle=\sum_{P,Q}\widehat{U}(P,Q)^{*}\bigg{(}2^{m}\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)2^{n}\left(\widehat{x_{1}}(P)-\widehat{y_{1}}(P)\right)+2^{n}\left(\widehat{y_{1}}(P)-\widehat{x_{1}}(P)\right)2^{m}\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\bigg{)}$
$\displaystyle=2^{n+m}\sum_{P,Q}\widehat{U}(P,Q)^{*}\bigg{(}\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\left(\widehat{x_{1}}(P)-\widehat{y_{1}}(P)\right)-\left(\widehat{x_{1}}(P)-\widehat{y_{1}}(P)\right)\left(\widehat{x_{2}}(Q)-\widehat{y_{2}}(Q)\right)\bigg{)}$
$\displaystyle=0.$

$\square$

See 9

#####  Proof.

In Lemma 12, we proved that $\mathcal{F}$ is linear, which implies that for
any $Z,Z^{\prime}\in\mathcal{C}$,

$\displaystyle\mathcal{F}(Z)-\mathcal{F}(Z^{\prime})=\mathcal{F}(Z-Z^{\prime}).$

In order to prove that $\mathcal{F}$ is Lipschitz continuous, we simply need
to prove that there exists a $\gamma\in\mathbb{R}$ such that, for all $Z\neq
Z^{\prime}$,101010In the case that $Z=Z^{\prime}$, trivially $Z-Z^{\prime}=0$
and $\mathcal{F}(Z-Z^{\prime})=0$, meaning any value for $\gamma$ suffices.

$\displaystyle\frac{||\mathcal{F}(Z-Z^{\prime})||_{F}}{||Z-Z^{\prime}||_{F}}\leq\gamma.$

Letting $Y=Z-Z^{\prime}$, this is equivalent to proving

$\displaystyle\sup_{Y\neq
0}\frac{||\mathcal{F}(Y)||_{F}}{||Y||_{F}}\leq\gamma.$ (169)

Vectorizing $Y$, as a vector of size $|\mathcal{C}|$, the Frobenius norm can
be expressed as

$\displaystyle||Y||_{F}=\sqrt{\sum_{i}|Y_{i}|^{2}}.$

Furthermore, since $\mathcal{F}$ is a linear operator, it maps $Y$ to some new
joint state $Y^{\prime}=\mathcal{F}(Y)\in\mathcal{C}$, with entries

$\displaystyle Y^{\prime}_{j}=\sum_{i}c_{ij}Y_{i}.$

for some linear coefficients $c_{ij}\in\mathbb{C}$. Note that since the
setting is restricted to finite-valued quantum zero-sum games, with a finite-
valued utility function $\mathcal{U}$, the coefficients $c_{ij}$ are
guaranteed to be finite-valued. Therefore, the right-hand side of Equation 169
can be re-expressed as

$\displaystyle\sup_{Y\neq 0}\frac{||\mathcal{F}(Y)||_{F}}{||Y||_{F}}$
$\displaystyle=\sup_{Y\neq 0}\frac{||Y^{\prime}||_{F}}{||Y||_{F}}=\sup_{Y\neq
0}\sqrt{\frac{\sum_{j}|\sum_{i}c_{ij}Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}.$

Letting $c^{*}=\max(\\{c_{i,j}\\})$,

$\displaystyle\sup_{Y\neq
0}\sqrt{\frac{\sum_{j}|\sum_{i}c_{ij}Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}\leq\sup_{Y\neq
0}\sqrt{\frac{\sum_{j}|\sum_{i}c^{*}Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}=\sup_{Y\neq
0}\sqrt{\frac{c^{*}\sum_{j}|\sum_{i}Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}$

and leveraging the triangle inequality,

$\displaystyle\sup_{Y\neq
0}\sqrt{\frac{c^{*}\sum_{j}|\sum_{i}Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}\leq\sup_{Y\neq
0}\sqrt{\frac{c^{*}\sum_{j}\sum_{i}|Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}=\sup_{Y\neq
0}\sqrt{\frac{c^{*}|\mathcal{C}|\cdot\sum_{i}|Y_{i}|^{2}}{\sum_{i}|Y_{i}|^{2}}}=\sqrt{c^{*}|\mathcal{C}|}.$

Thus, for any $\gamma\geq\sqrt{c^{*}|\mathcal{C}|}$, Equation 169 is
satisfied. Recall that $c^{*}=\Theta(1)$ for games in $[-1,1]$. Therefore, the
linear operator $\mathcal{F}$ is Lipschitz continuous where
$\gamma_{F,F}=\mathcal{O}(\sqrt{4^{d}})$. Applying similar argumentation for
the case of real vector and norm pairs $(\ell_{\infty},\ell_{1})$, we get
$\gamma_{\infty,1}=\mathcal{O}(1)$. Indeed, it is easy to see

$\displaystyle\sup_{Y\neq 0}\frac{||\mathcal{F}(Y)||_{\infty}}{||Y||_{1}}$
$\displaystyle=\sup_{Y\neq
0}\frac{||Y^{\prime}||_{\infty}}{||Y||_{1}}=\sup_{Y\neq
0}\sqrt{\frac{\max_{j}|\sum_{i}c_{ij}Y_{i}|}{\sum_{i}|Y_{i}|}}\leq
c^{*}=\mathcal{O}(1).$

$\square$

See 1

#####  Proof.

Denote a strong solution, satisfying Equation 20, as $\Psi^{S}$. Since
$\mathcal{F}$ is monotone, from the definition of monotonicty,
$\forall\>\Psi\in\mathcal{C}$,

$\displaystyle\textnormal{Tr}\left[{(\mathcal{F}(\Psi)-\mathcal{F}(\Psi^{S}))}\>{(\Psi-\Psi^{S})}\right]\geq
0,$ (170)

which implies that,

$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi)}\>{(\Psi-\Psi^{S})}\right]-\textnormal{Tr}\left[{\mathcal{F}(\Psi^{S})}\>{(\Psi-\Psi^{S})}\right]\geq
0.$ (171)

Since $\Psi^{S}$ is a strong solution, by Equation 20,

$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi)}\>{(\Psi-\Psi^{S})}\right]\geq\textnormal{Tr}\left[{\mathcal{F}(\Psi^{S})}\>{(\Psi-\Psi^{S})}\right]\geq
0.$ (172)

Therefore, $\forall\>\Psi\in\mathcal{C}$,

$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi)}\>{(\Psi^{S}-\Psi)}\right]\leq
0,$ (173)

meaning $\Psi^{S}$ satisfies Equation 21 and is, thus, also a weak solution.
$\square$

See 2

#####  Proof.

Denote a weak solution, satisfying Equation 21, as $\Psi^{W}$. In Lemma 10 we
proved that spectraplexes are convex, which implies that $\mathcal{A}$,
$\mathcal{B}$, and, thus, $\mathcal{C}$ are convex sets. Therefore, by the
definition of convex set, for any $\Psi\in\mathcal{C}$ and $t\in(0,1]$,

$\displaystyle\Psi^{W}+t\cdot(\Psi-\Psi^{W})\in\mathcal{C}.$ (174)

Plugging this state into Equation 21 gives

$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi^{W}+t\cdot(\Psi-\Psi^{W}))}\>{\Big{(}\Psi^{W}+t\cdot(\Psi-\Psi^{W})-\Psi^{W}\Big{)}}\right]\geq
0$ (175) $\displaystyle
t\cdot\textnormal{Tr}\left[{\mathcal{F}(\Psi^{W}+t\cdot(\Psi-\Psi^{W}))}\>{\Big{(}\Psi-\Psi^{W}\Big{)}}\right]\geq
0$ (176)
$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi^{W}+t\cdot(\Psi-\Psi^{W}))}\>{\Big{(}\Psi-\Psi^{W}\Big{)}}\right]\geq
0.$ (177)

Since $\mathcal{F}$ is Lipschitz continuous, we can take the following limit,

$\displaystyle\lim_{t\rightarrow
0}\textnormal{Tr}\left[{\mathcal{F}(\Psi^{W}+t\cdot(\Psi-\Psi^{W}))}\>{\Big{(}\Psi-\Psi^{W}\Big{)}}\right]\geq
0.$ (178)

The resulting expression,

$\displaystyle\textnormal{Tr}\left[{\mathcal{F}(\Psi^{W})}\>{\Big{(}\Psi-\Psi^{W}\Big{)}}\right]\geq
0,\>\>\forall\Psi\in\mathcal{C},$ (179)

implies that $\Psi^{W}$ satisfies Equation 20 and is also a strong solution.
$\square$

######  Lemma 12 (Linearity of $\mathcal{F}$)

$\mathcal{F}:\mathcal{A}\times\mathcal{B}\mapsto\mathcal{C}$ is a linear
operator. Therefore, $\mathcal{F}$ must satisfy the following properties:

  1. 1.

For any $\lambda\in\mathbb{C}$,
$\mathcal{F}(\lambda\alpha,\lambda\beta)=\lambda\mathcal{F}(\alpha,\beta)$.

  2. 2.

For $\alpha,\alpha^{\prime}\in\mathcal{A}$ and
$\beta,\beta^{\prime}\in\mathcal{B}$,
$\mathcal{F}(\alpha+\alpha^{\prime},\beta+\beta^{\prime})=\mathcal{F}(\alpha,\beta)+\mathcal{F}(\alpha^{\prime},\beta^{\prime})$.

#####  Proof.

From Equation 16 and Equation 17, it follows that

$\displaystyle\mathcal{F}_{\alpha}(\alpha,\beta)$
$\displaystyle=\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\beta)\right]=\mathcal{F}_{\alpha}(\beta)\in\mathcal{A}$
$\displaystyle\mathcal{F}_{\beta}(\alpha,\beta)$
$\displaystyle=-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\alpha\otimes\mathbb{1}_{m})\right]=\mathcal{F}_{\beta}(\alpha)\in\mathcal{B}$
$\displaystyle\mathcal{F}(\alpha,\beta)$
$\displaystyle=(\mathcal{F}_{\alpha}(\alpha,\beta),\mathcal{F}_{\beta}(\alpha,\beta))\in\mathcal{C}$

such that $\mathcal{F}:\mathcal{A}\times\mathcal{B}\mapsto\mathcal{C}$.

We will now prove that $\mathcal{F}$ satisfies the _multiplicative property_.
If $\lambda\in\mathbb{C}$, then

$\displaystyle\mathcal{F}_{\lambda\alpha}(\lambda\alpha,\lambda\beta)$
$\displaystyle=\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\lambda\beta)\right]=\lambda\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\beta)\right]=\lambda\mathcal{F}_{\alpha}(\alpha,\beta)$
$\displaystyle\mathcal{F}_{\lambda\beta}(\lambda\alpha,\lambda\beta)$
$\displaystyle=-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\lambda\alpha\otimes\mathbb{1}_{m})\right]=-\lambda\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\alpha\otimes\mathbb{1}_{m})\right]=\lambda\mathcal{F}_{\beta}(\alpha,\beta),$

which implies, as desired,

$\displaystyle\mathcal{F}(\lambda\alpha,\lambda\beta)=\Big{(}\mathcal{F}_{\lambda\alpha}(\lambda\alpha,\lambda\beta),\mathcal{F}_{\lambda\beta}(\lambda\alpha,\lambda\beta)\Big{)}=\Big{(}\lambda\mathcal{F}_{\alpha}(\alpha,\beta),\lambda\mathcal{F}_{\beta}(\alpha,\beta)\Big{)}=\lambda\mathcal{F}(\alpha,\beta).$

We will now prove that $\mathcal{F}$ satisfies the _additive property_. If
$\alpha,\alpha^{\prime}\in\mathcal{A}$ and
$\beta,\beta^{\prime}\in\mathcal{B}$, then

$\displaystyle\mathcal{F}_{\alpha+\alpha^{\prime}}(\alpha+\alpha^{\prime},\beta+\beta^{\prime})$
$\displaystyle=\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes(\beta+\beta^{\prime}))\right]$
$\displaystyle=\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\beta)+U^{\dagger}(\mathbb{1}_{n}\otimes\beta^{\prime})\right]$
$\displaystyle=\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\beta)\right]+\textnormal{Tr}_{\mathcal{B}}\left[U^{\dagger}(\mathbb{1}_{n}\otimes\beta^{\prime})\right]$
$\displaystyle=\mathcal{F}_{\alpha}(\alpha,\beta)+\mathcal{F}_{\alpha^{\prime}}(\alpha^{\prime},\beta^{\prime})$
$\displaystyle\mathcal{F}_{\beta+\beta^{\prime}}(\alpha+\alpha^{\prime},\beta+\beta^{\prime})$
$\displaystyle=-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}((\alpha+\alpha^{\prime})\otimes\mathbb{1}_{m})\right]$
$\displaystyle=-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\alpha\otimes\mathbb{1}_{m})+U^{\dagger}(\alpha^{\prime}\otimes\mathbb{1}_{m})\right]$
$\displaystyle=-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\alpha\otimes\mathbb{1}_{m})\right]-\textnormal{Tr}_{\mathcal{A}}\left[U^{\dagger}(\alpha^{\prime}\otimes\mathbb{1}_{m})\right]$
$\displaystyle=\mathcal{F}_{\beta}(\alpha,\beta)+\mathcal{F}_{\beta^{\prime}}(\alpha^{\prime},\beta^{\prime}),$

which implies, as desired,

$\displaystyle\mathcal{F}(\alpha+\alpha^{\prime},\beta+\beta^{\prime})$
$\displaystyle=\Big{(}\mathcal{F}_{\alpha}(\alpha,\beta)+\mathcal{F}_{\alpha^{\prime}}(\alpha^{\prime},\beta^{\prime}),\mathcal{F}_{\beta}(\alpha,\beta)+\mathcal{F}_{\beta^{\prime}}(\alpha^{\prime},\beta^{\prime})\Big{)}$
$\displaystyle=\Big{(}\mathcal{F}_{\alpha}(\alpha,\beta),\mathcal{F}_{\beta}(\alpha,\beta)\Big{)}+\Big{(}\mathcal{F}_{\alpha^{\prime}}(\alpha^{\prime},\beta^{\prime}),\mathcal{F}_{\beta^{\prime}}(\alpha^{\prime},\beta^{\prime})\Big{)}$
$\displaystyle=\mathcal{F}(\alpha,\beta)+\mathcal{F}(\alpha^{\prime},\beta^{\prime}).$

$\square$

##  Appendix D Convergence Analysis Proofs

###### Proposition 7

If $g(X)$ is a $\mu$-strongly convex function and $h(X)$ is an
$\alpha$-strongly convex function, then $f(X)=g(X)+h(X)$ is a
$\mu+\alpha$-strongly convex function.

#####  Proof.

By the defintion of strong convexity (Definition 2), $\forall
X,Y\in\mathcal{C}$

$\displaystyle g(X)\geq g(Y)+\langle\nabla
g(Y),X-Y\rangle+\frac{\mu}{2}\|X-Y\|_{F}^{2},$ (180) $\displaystyle h(X)\geq
h(Y)+\langle\nabla h(Y),X-Y\rangle+\frac{\alpha}{2}\|X-Y\|_{F}^{2}.$ (181)

Summing together these two expressions gives

$\displaystyle g(X)+h(X)\geq g(Y)+h(Y)+\langle\nabla g(Y)+\nabla
h(Y),X-Y\rangle+\frac{\mu+\alpha}{2}\|X-Y\|_{F}^{2}$ (182)
$\displaystyle\implies f(X)\geq f(Y)+\langle\nabla
f(Y),X-Y\rangle+\frac{\mu+\alpha}{2}\|X-Y\|_{F}^{2},$ (183)

which implies $f$ is $(\mu+\alpha)$-strongly convex. $\square$

See 3

#####  Proof.

To prove that $\phi_{t}(Z)$ is $\frac{\mu}{\eta}$-strongly convex, we will
leverage Proposition 7. Namely, by proving that
$\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(Z-\Phi_{t-1})}\right]$ is
convex (i.e., $0$-strongly convex) and that
$\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ is $\frac{\mu}{\eta}$-strongly
convex, then

$\displaystyle\phi_{t}(Z)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(Z-\Phi_{t-1})}\right]+\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1}),$
(184)

must be $0+\frac{\mu}{\eta}=\frac{\mu}{\eta}$-strongly convex.

We begin by proving that $\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ is
$\frac{\mu}{\eta}$-strongly convex. From the definition of strong convexity
(Definition 2), if $h$ is $\mu$-strongly convex, then the Bregman divergence
$\mathcal{BD}_{h}$ must also be $\mu$-strongly convex. This implies that

$\displaystyle\frac{1}{\eta}\mathcal{BD}_{h}(X\|Y)\geq\frac{\mu}{2\eta}\|X-Y\|_{F}^{2},\hskip
7.22743pt\forall X,Y\in\mathcal{C},$ (185)

meaning the function $\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ must be
$\frac{\mu}{\eta}$-strongly convex.

We now prove that
$f(Z)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(Z-\Phi_{t-1})}\right]$
is convex. For all $X,Y\in\mathcal{C}$,

$\displaystyle
f(X)-f(Y)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(X-\Phi_{t-1})}\right]-\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(Y-\Phi_{t-1})}\right]=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(X-Y)}\right].$
(186)

Since $\forall Z\in\mathcal{C},\>\nabla f(Z)=\mathcal{F}(\Psi_{t})$, this
implies that

$\displaystyle
f(X)-f(Y)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t})}\>{(X-Y)}\right]=\textnormal{Tr}\left[{\nabla
f(Y)}\>{(X-Y)}\right],$ (187)

meaning, by Definition 1, $f$ is convex. $\square$

See 4

#####  Proof.

To prove that $\psi_{t}(Z)$ is $\frac{\mu}{\eta}$-strongly convex, we will
leverage Proposition 7. Namely, by proving that
$\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(Z-\Phi_{t-1})}\right]$ is
convex (i.e. $0$-strongly convex) and that
$\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ is $\frac{\mu}{\eta}$-strongly
convex, then

$\displaystyle\psi_{t}(Z)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(Z-\Phi_{t-1})}\right]+\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1}),$
(188)

must be $0+\frac{\mu}{\eta}=\frac{\mu}{\eta}$-strongly convex.

We begin by proving that $\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ is
$\frac{\mu}{\eta}$-strongly convex. From the definition of strong convexity
(Definition 2), if $h$ is $\mu$-strongly convex, then the Bregman divergence
$\mathcal{BD}_{h}$ must also be $\mu$-strongly convex. This implies that

$\displaystyle\frac{1}{\eta}\mathcal{BD}_{h}(X\|Y)\geq\frac{\mu}{2\eta}\|X-Y\|_{F}^{2},\hskip
7.22743pt\forall X,Y\in\mathcal{C},$ (189)

meaning the function $\frac{1}{\eta}\mathcal{BD}_{h}(Z\|\Phi_{t-1})$ must be
$\frac{\mu}{\eta}$-strongly convex.

We now prove that
$f(Z)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(Z-\Phi_{t-1})}\right]$
is convex. For all $X,Y\in\mathcal{C}$,

$\displaystyle
f(X)-f(Y)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(X-\Phi_{t-1})}\right]-\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(Y-\Phi_{t-1})}\right]=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(X-Y)}\right].$
(190)

Since $\forall Z\in\mathcal{C},\>\nabla f(Z)=\mathcal{F}(\Psi_{t})$, this
implies that

$\displaystyle
f(X)-f(Y)=\textnormal{Tr}\left[{\mathcal{F}(\Psi_{t-1})}\>{(X-Y)}\right]=\textnormal{Tr}\left[{\nabla
f(Y)}\>{(X-Y)}\right],$ (191)

meaning, by Definition 1, $f$ is convex. $\square$


# Adversarial Attacks and Defense Methods for Power Quality Recognition

Jiwei Tian, Buhong Wang, , Jing Li, Zhen Wang and Mete Ozay An early version
of this work was presented in part at the 2019 IEEE SmartGridComm [1]. This
work was supported by the National Natural Science Foundation of China under
Grant 61902426 and the National University of Defense Technology Research Fund
under Grant ZK21-44. Jiwei Tian is with Air Traffic Control and Navigation
College, Air Force Engineering University, Xi’an, China (e-mail:
tianjiwei2016@163.com). B. Wang and Z. Wang are with Information and
Navigation College, Air Force Engineering University, Xi’an, China. J. Li is
with Henan University of Technology, Zhengzhou, China.

###### Abstract

Vulnerability of various machine learning methods to adversarial examples has
been recently explored in the literature. Power systems which use these
vulnerable methods face a huge threat against adversarial examples. To this
end, we first propose a signal-specific method and a universal signal-agnostic
method to attack power systems using generated adversarial examples. Black-box
attacks based on transferable characteristics and the above two methods are
also proposed and evaluated. We then adopt adversarial training to defend
systems against adversarial attacks. Experimental analyses demonstrate that
our signal-specific attack method provides less perturbation compared to the
FGSM (Fast Gradient Sign Method), and our signal-agnostic attack method can
generate perturbations fooling most natural signals with high probability.
What’s more, the attack method based on the universal signal-agnostic
algorithm has a higher transfer rate of black-box attacks than the attack
method based on the signal-specific algorithm. In addition, the results show
that the proposed adversarial training improves robustness of power systems to
adversarial examples.

###### Index Terms:

Adversarial example, adversarial training, universal perturbation.

##  I Introduction

Power quality refers to a variety of electromagnetic phenomena that
characterize voltage and current measured at a given time instance and
location in a power system [2]. Disturbance of power quality (PQ) signals can
cause severe problems in electrical grids [3]. PQ disturbances occurring in
grids may cause financial loss, damage and faulty operation of equipment
installed in grids and end-user devices [4]. In end-user systems, PQ
disturbances can cause loss of data, memory failures of sensitive loads (such
as computers, protection and relay devices), and unstable operation of
controls [5]. Therefore, if PQ disturbances are not properly recognized and
mitigated by monitoring systems, then power grid assets can be harmed
resulting in high costs and safety hazards in extreme but realistic cases.
Numerous techniques have been proposed to address the aforementioned problems
[4, 6]. Among them, deep neural networks (DNNs) have been employed to improve
accuracy for recognition of PQ disturbances replacing hand-crafted features by
learned feature representations [7].

Although DNNs show outstanding performance in various tasks, they are
vulnerable to a form of attacks called “adversarial example” [8]. Adversarial
examples with different attributes have been explored for image, text, and
speech processing in [8, 9, 10]. DNNs have been recently used in cyber
physical systems (CPSs) [11], but exploration and assessment of their security
is still an open problem. Security risks caused by adversarial machine
learning (ML) methods in CPSs have been discussed in [12]. Vulnerabilities of
ML algorithms utilized in power systems were first shown in [13]. Additional
works have addressed security problems for grid event classification [14], N-1
security [15], power quality signal classification [13] and detection of false
data injection attacks [16, 17]. The vulnerabilities of load forecasting
algorithms caused by perturbations injected into input temperatures were
studied in [18]. A domain-specific framework was proposed to evaluate security
and resilience of load forecasting algorithms in [19] and experimental results
showed that DNN-based load predictions may suffer from worst-case attacks even
with a partial control of network. From a defense perspective, adversarial
training [20] and game theory [21] are respectively adopted to design
resilient load forecasting systems. Adversarial attacks for DNN-based state
estimation were studied in [22].

Characterization of these attacks and defense mechanisms is an open problem
that has not been fully explored and understood. An adversarial signal attack
is a vital threat, since signals generated by the attack are perceptually
similar to the original signals. Therefore, a human operator of the system may
not be aware of the presence of the attack. Then, the system may take a wrong
action according to the incorrectly predicted signal class affecting the
physical system. Vulnerability of DNNs against particular adversarial attacks
in power systems was analyzed in [13, 1]. Among them, a variant of FGSM attack
[23] is proposed to generate adversarial examples for PQ classification in
[13]. A more accurate and computationally efficient attack method and
corresponding defense method are proposed in [1]. However, stronger and more
practical attack methods (such as as universal perturbations and black-box
attacks) and corresponding defense strategies need to be explored more
comprehensively, which is lacking in the current literature.

In order to defend power systems against these attacks and improve their
robustness, learning models deployed in the systems should be carefully
analyzed. In this paper, we explore adversarial attacks to CNN (Convolutional
Neural Network) algorithms and examine the corresponding defense methods for
PQ recognition tasks. The main contributions of this work are summarized as
follows111The code used for reproducing the experimental analyses and results
are provided on https://github.com/JiweiTian/AT_SG.git.:

  * •

We propose a signal-specific algorithm which outperforms a state-of-the-art
method [13] for generating adversarial perturbation of signals in power
systems.

  * •

We analyze existence of universal signal-agnostic perturbations generated by
CNNs in power systems, and propose an algorithm to recognize such
perturbations. Our results show that generated universal perturbations make
models discriminate natural power signals222We consider power signals commonly
measured by a particular system operator of a power system as natural power
signals. into several statistically dominant signal categories.

  * •

We explore and analyze black-box attacks: the attacker can train a local
substitute model and generate adversarial samples based on the substitute
model, and then use its transferable characteristics to attack the target
model. The experimental results show that the attack method based on the
universal signal-agnostic algorithm has a higher transfer rate of black-box
attacks than the attack method based on the signal-specific algorithm, which
is a huge and practical threat for CNN based PQ classification.

  * •

We propose using adversarial training as a defense method to improve
robustness of learning models employed in power systems to adversarial
attacks.

  * •

We quantitatively analyze misclassification of different attacks, and
summarize the characteristics of different attack methods for PQ signals. We
suggest an approach to qualitatively analyze the relationship between the
characteristics of adversarial attacks to PQ signals and defense methods in
the Smart Grid. For this purpose, we visualize signal measurements and their
feature representations learned at different layers of CNNs using the t-SNE
algorithm [24].

The paper is organized as follows. In Section II, we provide an overview of
adversarial methods and describe our proposed attack algorithms. In Section
III, our proposed adversarial training method is introduced. Experimental
analyses of the proposed methods are given for various case studies in Section
IV. Conclusion and discussion are provided in Section V.

##  II Adversarial Attacks for Power Quality Classification using CNNs

Figure 1: Adversarial attacks on power signal recognition algorithms using
deep learning methods in the Smart Grid.

###  II-A Attack Models and Objectives

As shown in Fig. 1, a well-trained CNN can be used for power quality
assessment in the Smart Grid, which is crucial for its robust operation.
However, an attacker can penetrate a communication network of the system, and
use adversarial signals instead of original signals to attack the system to
fool the CNN for misclassification of the PQ signals (for example, Man-In-The-
Middle attack). Note that we consider attacks in the testing phase
(adversarial examples and adversarial attacks) rather than in the training
phase (poisoning attacks). The objective of the attacker is generating
imperceptible adversarial signals similar to the original signals, resulting
in their misclassification by algorithms without being noticed by human
operators333Suppose that the original signal belongs to Class 1. Since the
added perturbation is so small (imperceptible), it should belong to the same
type as the original signal from the human operator’s point of view (Class 1).
However, the target CNN will regard it as a new type (for example, Class 2).

In our white-box attack model, we assume that attackers can access parameters
of CNNs deployed in power systems and use this knowledge to construct
adversarial signals. In practice, these parameters can be obtained by insider
threats or using various attack methods, such as eavesdropping on network
traffic or database intrusion. Rouhani et al. [25] suggest that white-box
attacks represent the most powerful attacker that can appear in real-world
settings. Although these assumptions render a strong attacker that may not
always represent the practical cases, they enable us to evaluate the
robustness and vulnerabilities under the worst-case scenario, providing an
upper bound on the impact of proposed attacks on PQ CNN models.

Although many adversarial attack methods have been proposed in image domain,
there are significant differences between PQ recognition tasks using power
signals and object recognition tasks using two or three dimensional images:

  * •

Data Characteristics: A PQ signal is a time-series, whose characteristics are
different from that of two or three dimensional images. Therefore, in order to
produce imperceptible PQ perturbations, the characteristics of time-series
data should be considered to avoid occurrence of abnormal values with great
changes. Besides, although image pixels take values from a fixed interval,
such as [0,1] or [0,255], PQ signals have no definite range.

  * •

Levels of Perturbation: In image processing and computer vision tasks, norms
such as $\ell_{0}$, $\ell_{2}$ and $\ell_{\infty}$ are used to model
adversarial perturbations. Since PQ signals have no definite range, some
adversarial attack methods based on box constraints [26] and “change of
variables” such as C$\&W$ [27] are not suitable for PQ signals. Although
$\ell_{0}$ norm-based perturbations such as Sparsefool [28] and “One pixel
attack” [29] in image domain will not affect human perception, the outliers
generated by the $\ell_{0}$ norm-based perturbations will have a greater
impact on time-series PQ signals. Besides, for PQ signals, it’s hard to
determine a maximum budget for $\ell_{\infty}$ norm-based perturbations (the
maximum budget will affect human perception of PQ signals, and each value is
allowed to be changed by up to this limit, with no limit on the number of
input data that are modified). Therefore, we consider the $\ell_{2}$ norm case
for PQ signals in the paper.

Although FGSM was employed in particular power systems444Therefore, we
consider the FGSM method as the baseline and state-of-the-art for adversarial
attacks in PQ recognition in our experimental analyses. [13], development of a
general and suitable approach for adapting adversarial attacks for PQ
recognition in power systems is an open problem. We address this problem by
proposing two methods called signal-specific and signal-agnostic attacks. We
realize these approaches to generate adversarial signals for PQ recognition by
adapting DeepFool [30] and universal attack methods in power systems. Although
other adversarial attack methods have been proposed in image domain, many
methods and their variants are not suitable for the PQ signal field for
various reasons, such as the existence of box constraints (L-BFGS [26], C$\&W$
[27]) and artificially determined attack parameters (FGSM [23], I-FGSM [31],
JSMA [32]). We consider that our proposed approaches can be employed to adapt
a wider class of attack models for various tasks in power systems by
researchers, especially attack methods based on geometrical perspective [33]
(such as targeted DeepFool [34][35], DeepFool-5 [36] and so on).

###  II-B Signal-Specific Adversarial Signals

In this section, we introduce our signal-specific adversarial attack method
which is proposed to generate adversarial perturbations of signals in power
systems.

Suppose that $f:\mathbb{R}^{n}\to\mathbb{R}^{K}$ is a classifier computed by a
learning model to determine the class label of a signal
$\boldsymbol{x}_{0}\in\mathbb{R}^{n}$ among $K$ power quality classes. In this
work, deep learning algorithms are used to compute feature representations of
PQ signals and classifiers $f$. Therefore, we will identify the classifier by
a non-linear function $f(\boldsymbol{x}_{0}:\mathcal{W})$ which is
parameterized by a set of tensors $\mathcal{W}$ denoting weights and bias
vectors called model parameters of CNNs. The parameters $\mathcal{W}$ are
optimized using a stochastic optimization algorithm such as stochastic
gradient descent. Since the target model in this paper is fixed, for the sake
of simplicity, we don’t show the dependence of $f$ on $\mathcal{W}$. The class
label of the power signal $\boldsymbol{x}_{0}$ is estimated by

$\hat{y}\left(\boldsymbol{x}_{0}\right)=\operatorname*{arg\,max}\limits_{k}f_{k}(\boldsymbol{x}_{0}),$
(1)

where $f_{k}(\boldsymbol{x}_{0})$ is the output of $f(\boldsymbol{x}_{0})$ for
the $k^{th}$ class. An adversarial perturbation is defined as the minimal
(imperceptible) perturbation $\boldsymbol{r}$ that is sufficient to change the
estimated power quality label $\hat{y}\left(\boldsymbol{x}_{0}\right)$ at
$\boldsymbol{x}_{0}$ by

$\displaystyle\Delta(\boldsymbol{x}_{0};\hat{y}(\boldsymbol{x}_{0}))$
$\displaystyle\triangleq\min_{\boldsymbol{r}}\|\boldsymbol{r}\|_{2}$ (2)
$\displaystyle{\rm subject\;to}\;\quad\hat{y}(\boldsymbol{x}_{0}$
$\displaystyle+\boldsymbol{r})\neq\hat{y}(\boldsymbol{x}_{0}),$

where $\|\cdot\|_{2}$ denotes the $\ell_{2}$ norm.

In adversarial attacks, attackers can make learned models incorrectly estimate
class labels of attacked power signals by adding the minimal imperceptible
perturbation computed by (2). For instance, normal power signals can be
misclassified as impulse signals, and vice versa. Thereby, adversarial attacks
can affect power quality monitoring systems by adversarial data injection, and
cause failure of their operation. The minimal perturbation computed using (2)
for different classes and samples of PQ signals may be different from each
other. To generate minimum perturbation of the signal ${\boldsymbol{x}_{0}}$
optimizing the closest decision boundary of $f$ from $\boldsymbol{x}_{0}$, a
signal-specific adversarial attack method (SSA) is proposed in Algorithm 1.

CNNs partition high dimensional class regions with nonlinear decision
boundaries by locally piecewise linear functions implemented in network nodes
into subregions [37] which are convex polytopes [38]. Wong and Zolter [39]
propose that if CNNs are optimized to minimize the worst case loss over convex
outer approximation of the set of activations reachable through a norm-bounded
perturbation (called adversarial polytope), then the trained CNN models is
provably robust to any norm-bounded adversarial attack. We employ this result
to generate adversarial signals according to a convex outer approximation of
an adversarial polytope of trained CNN models. For this purpose, we first
define the region (polytope) of the space of adversarial signals by

$\mathcal{P}\triangleq\bigcap\limits_{k=1}^{K}\\{\boldsymbol{x}:f_{\hat{y}\left(\boldsymbol{x}_{0}\right)}\left(\boldsymbol{x}_{0}\right)\geq
f_{k}\left(\boldsymbol{x}\right)\\}.$ (3)

Activations of the model for the signals $\boldsymbol{x}$ belonging to this
region for each different $k^{th}$ class are bounded by that for the signals
$\boldsymbol{x}_{0}$ for the predicted class
$\hat{y}\left(\boldsymbol{x}_{0}\right)$. Next, we generate adversarial
signals in Algorithm 1 as follows:

  * •

Step 4 and 5: The region $\mathcal{P}$ is approximated at each $i^{th}$ epoch
by

$\hat{\mathcal{P}}_{i}=\bigcap\limits_{k=1}^{K}\\{\boldsymbol{x}:f_{k}^{{}^{\prime}}+(\boldsymbol{w}_{k}^{\prime})^{T}\boldsymbol{x}\leq
0\\},$ (4)

where $\boldsymbol{w}_{k}^{\prime}=\nabla
f_{k}\left(\boldsymbol{x}_{i}\right)^{T}-\nabla
f_{\hat{y}\left(\boldsymbol{x}_{0}\right)}\left(\boldsymbol{x}_{i}\right)^{T}$
(at step 4) and
$f_{k}^{\prime}=f_{k}\left(\boldsymbol{x}_{i}\right)-f_{\hat{y}\left(\boldsymbol{x}_{0}\right)}\left(\boldsymbol{x}_{i}\right)$
(at step 5). $\boldsymbol{w}_{k}^{\prime}$ denotes the difference between
gradients of functions $f$ of $\boldsymbol{x}_{i}$ for the $k^{th}$ class and
for the predicted class $\hat{y}\left(\boldsymbol{x}_{0}\right)$ with respect
to model parameters.

  * •

Step 7: We compute a convex outer approximation (boundary) of the polytope and
generate signals far from this boundary. Therefore, first the closest
hyperplane $\hat{l}$ of the boundary of $\mathcal{P}$ is computed where
$|\cdot|$ denotes the absolute value.

  * •

Step 8: Next, the minimum perturbation
$\boldsymbol{r}_{*}\left(\boldsymbol{x}_{0}\right)$ projecting
$\boldsymbol{x}_{0}$ on $\hat{l}$ is approximated at each $i^{th}$ epoch by
$\boldsymbol{r}_{i}$.

  * •

Step 9 and 10: Then, the adversarial signal $\boldsymbol{x}_{i}$ and epoch
variable $i$ are updated at the step 9 and 10, respectively.

  * •

The steps 3-10 are iterated while estimated class labels of the normal
$\boldsymbol{x}_{0}$ and the adversarial signal $\boldsymbol{x}_{i}$ are same.

  * •

Step 12: Finally, the algorithm injects the sum of perturbations
$\boldsymbol{r}_{j},j=1,2,\ldots,i$ to the original signal to compute the
adversarial PQ signal.

Algorithm 1 Generating Signal-Specific Adversarial Signals.

0: $\boldsymbol{x}$: An input power quality signal, $f$: A learned signal
classification model.

0: Adversarial power quality signal $\hat{\boldsymbol{x}}$.

1: Initialization: $i\leftarrow 0$ and
$\boldsymbol{x}_{0}\leftarrow\boldsymbol{x}$.

2: while
$\hat{y}\left(\boldsymbol{x}_{i}\right)=\hat{y}\left(\boldsymbol{x}_{0}\right)$
do

3: for $k\neq\hat{y}\left(\boldsymbol{x}_{0}\right)$ do

4: $\boldsymbol{w}_{k}^{\prime}\leftarrow\nabla
f_{k}\left(\boldsymbol{x}_{i}\right)-\nabla
f_{\hat{y}\left(\boldsymbol{x}_{0}\right)}\left(\boldsymbol{x}_{i}\right)$

5: $f_{k}^{\prime}\leftarrow
f_{k}\left(\boldsymbol{x}_{i}\right)-f_{\hat{y}\left(\boldsymbol{x}_{0}\right)}\left(\boldsymbol{x}_{i}\right)$

6: end for

7:
$\hat{l}\leftarrow\operatorname*{arg\,min}\limits_{k\neq\hat{y}\left(\boldsymbol{x}_{0}\right)}\frac{\left|f_{k}^{\prime}\right|}{\left\|\boldsymbol{w}_{k}^{\prime}\right\|_{2}}$

8:
$\boldsymbol{r}_{i}\leftarrow\frac{\left|f_{\hat{l}}^{\prime}\right|}{\left\|\boldsymbol{w}_{\hat{\imath}}^{\prime}\right\|_{2}^{2}}\boldsymbol{w}_{\hat{l}}^{\prime}$

9: $\boldsymbol{x}_{i+1}\leftarrow\boldsymbol{x}_{i}+\boldsymbol{r}_{i}$

10: $i\leftarrow i+1$

11: end while

12: return Adversarial power quality signal
$\boldsymbol{x}+\sum\limits_{j=1}^{i}\boldsymbol{r}_{j}$.

###  II-C Signal-Agnostic Adversarial Signals

In our proposed signal-agnostic adversarial attack method (SAA), attackers aim
to generate signal-agnostic and very small perturbation vectors that cause
normal PQ signals to be misclassified with high probability. An attacker can
make the PQ signal be misclassified with high probability by adding signal-
agnostic perturbations without spending time calculating the specific
perturbation, independent of class of actual PQ signal (e.g., normal, sag and
so on) fed to the system. Presence of such perturbations poses a great threat
to learning models in power systems, as the adversary may simply add the same
pre-computed signal-agnostic perturbation to the new signal and cause
misclassification. Therefore, pre-computed signal-agnostic perturbations are
called universal perturbations [40].

The goal of the SSA is to generate a quasi-imperceptible universal
perturbation signal vector $\boldsymbol{v}$ such that most PQ signals sampled
from a distribution $\mu$ are misclassified when applying the universal
perturbation signal. We compute the vector $\boldsymbol{v}$ considering the
following two constraints:

  1. 1.

The magnitude of the universal perturbation vector $\boldsymbol{v}$ is
controlled by a parameter $\xi>0$ using
$\lVert\boldsymbol{v}\rVert_{2}\leq\xi.$

  2. 2.

The misclassification probability is lower bounded for all signals
$\boldsymbol{x}$ sampled from the distribution $\mu$ by

$\underset{\boldsymbol{x}\sim\mu}{P}(\hat{y}(\boldsymbol{x}+\boldsymbol{v})\neq\hat{y}(\boldsymbol{x}))\geq
1-\delta,{\rm where}\;\delta\in(0,1).$ (5)

Let $\mathcal{X}=\left\\{\boldsymbol{x}_{i}\right\\}_{i=1}^{m}$ be a set of PQ
signals sampled from the distribution $\mu$. Our method aims at finding a
perturbation $\boldsymbol{v}$ fooling signals satisfying
$\lVert\boldsymbol{v}\rVert_{2}\leq\xi$. Our proposed algorithm (Algorithm 2)
gradually crafts adversarial perturbations on signals belonging to
$\mathcal{X}$. At each iteration, the minimal perturbation
$\Delta\boldsymbol{v}_{i}$ pushing the sample $\boldsymbol{x}_{i}$ to the
decision boundary is calculated and added to the current perturbation. More
precisely, if the current perturbation $\boldsymbol{v}$ does not fool the
sample signal $\boldsymbol{x}_{i}$, then we will compute an additional
perturbation $\Delta\boldsymbol{v}_{i}$ with minimal norm to fool the sample
signal $\boldsymbol{x}_{i}$ at step 5 of the Algorithm 2 by solving the
following optimization problem

$\displaystyle\Delta\boldsymbol{v}_{i}$
$\displaystyle=\operatorname*{arg\,min}\limits_{\boldsymbol{r}}\,\,\lVert\boldsymbol{r}\rVert_{2}$
(6) $\displaystyle{\rm subject\;to}$
$\displaystyle\quad\hat{y}\left(\boldsymbol{x}_{i}+\boldsymbol{v}+\boldsymbol{r}\right)\neq\hat{y}\left(\boldsymbol{x}_{i}\right).$

In order to meet the first constraint, the universal perturbation is projected
on an $\ell_{2}$ ball of radius $\xi$ and centered at 0 at step 6 of the
Algorithm 2 by

$\displaystyle\Pi_{2,\xi}\left(\boldsymbol{v}+\Delta\boldsymbol{v}_{i}\right)$
$\displaystyle=\operatorname*{arg\,min}\limits_{\boldsymbol{v}^{\prime}}\lVert\boldsymbol{v}+\Delta\boldsymbol{v}_{i}-\boldsymbol{v}^{\prime}\rVert_{2}$
(7) $\displaystyle{\rm subject\;to}$
$\displaystyle\quad\lVert\boldsymbol{v}^{\prime}\rVert_{2}\leq\xi.$

The fooling rate is defined for the perturbed training set $\mathcal{X}$ by

$R\left(\mathcal{X},\boldsymbol{v}\right):=\frac{1}{\sigma({\mathcal{X}})}\sum_{\boldsymbol{x}\in{\mathcal{X}}}{\mathbb{I}({\hat{y}\left(\boldsymbol{x}_{i}+\boldsymbol{v}\right)\neq\hat{y}\left(\boldsymbol{x}_{i}\right)}}),$
(8)

where $\mathbb{I}(\cdot)$ is an indicator function and $\sigma({\mathcal{X}})$
denotes the size of the perturbed training set $\mathcal{X}$. If the rate
$R\left(\mathcal{X},\boldsymbol{v}\right)$ exceeds $1-\delta$, then the
algorithm terminates. This method does not ensure that we can find universal
perturbations with large misclassification rates. However, in our experiments,
we can obtain universal perturbations with relatively high misclassification
rates using this method with limited signals.

Algorithm 2 Generating Signal-Agnostic Adversarial Signals.

0: $\mathcal{X}$: A set of power quality signals. $\xi$: An upper bound of the
$\ell_{2}$ norm of the perturbation. $\delta$: A parameter used to control the
fooling rate. $f$: A learned signal classification model.

0: A signal-agnostic perturbation vector $\boldsymbol{v}$.

1: Initialization: $\boldsymbol{v}\leftarrow\boldsymbol{0}$

2: while $R\left(\mathcal{X},\boldsymbol{v}\right)\leq 1-\delta$ do

3: for each signal $\boldsymbol{x}_{i}\in\mathcal{X}$ do

4: if $\hat{y}(\boldsymbol{x}+\boldsymbol{v})=\hat{y}(\boldsymbol{x})$ then

5: Compute the minimal perturbation $\Delta\boldsymbol{v}_{i}$ that pushes
${\boldsymbol{x}_{i}+\boldsymbol{v}}$ to the decision boundary by solving (6).

6: Update the perturbation by
$\boldsymbol{v}\leftarrow\Pi_{2,\xi}\left(\boldsymbol{v}+\Delta\boldsymbol{v}_{i}\right)$.

7: end if

8: end for

9: end while

10: return Signal-agnostic perturbation vector $\boldsymbol{v}$.

###  II-D Black-box Attacks Against CNN Based Power Quality Recognition

Without knowing the target model $f$, the attacker can train his own
substitute model $f^{{}^{\prime}}$ and craft adversarial examples against
$f^{{}^{\prime}}$. Due to the transferability of adversarial examples [41,
42], this kind of black-box attacks also has a certain success rate. If the
adversarial example generated against the model $f^{{}^{\prime}}$ can also
mislead the model $f$, we call this black-box attack successful.

Given the well-trained substitute model $f^{{}^{\prime}}$, the attacker can
use various attack methods to generate adversarial examples. In other words,
the attacker can carry out white-box attacks based on the substitute model
$f^{{}^{\prime}}$ to generate corresponding adversarial examples, then use
them to implement black-box attacks against the target model $f$. In the
following experiments, we will explore and analyze black-box attacks based on
the substitute model $f^{{}^{\prime}}$ and the above proposed algorithms
(Algorithm 1 and Algorithm 2).

##  III Defending Learning Models using Adversarial Training

Adversarial training is used to improve robustness of a learning model to
adversarial attacks. In this approach, a model is trained using a large number
of adversarial examples generated by different attack methods. The basic
requirement of this approach is employment of the strongest possible attack to
generate adversarial examples using (1) and (2). Adversarial training improves
robustness of DNNs to adversarial attacks [23, 43], provide regularization
[23] for DNNs and improve their classification accuracy [44]. In this paper,
we adopt adversarial training as our defense method to improve robustness of
PQ classification models in power systems. Our proposed adversarial training
method is summarized in Algorithm 3.

Algorithm 3 Defending Models using Adversarial Training.

0: $D_{tr}$: A training dataset of clean samples. $N_{iter}$: The number of
iterations used for training.

0: A robust power quality classification model $f_{\Theta}$, where $\Theta$ is
a set of learnable parameters of the model $f$. Training the model
$f_{\Theta}$ using a training dataset $D_{tr}$:

1: for $1,2,\ldots,N_{iter}$ do

2: Optimize $\Theta$ by training $f_{\Theta}$ using $D_{tr}$.

3: end for

4: Generate a set of adversarial signals $\tilde{\mathcal{X}}$ based on normal
signals $\mathcal{X}$ using Algorithm 1. Retraining the model $f_{\Theta}$
using the set of signals $\tilde{\mathcal{X}}$:

5: for $1,2,\ldots,N_{iter}$ do

6: Update $\Theta$ by training $f_{\Theta}$ using $\tilde{\mathcal{X}}$ and
$\mathcal{X}$.

7: end for

8: return A robust power quality classification model $f_{\Theta}$.

TABLE I: Comparison of the FGSM, SSA and SAA.

| $\hat{\rho}_{\mathrm{adv}}$ | Misclassification Rate | Average Time (sec.)
---|---|---|---
FGSM | $770\times 10^{-3}$ | 0.88 | $1.2\times 10^{-2}$
SSA | $5.0\times 10^{-3}$ | 0.98 | $2.2\times 10^{-1}$
SAA | $26.0\times 10^{-3}$ | 0.74 | $1.6\times 10^{-6}$

##  IV Experimental Analyses

###  IV-A Experimental Setup

We evaluate accuracy of the proposed attack and defense methods for power
quality assessment by classifying signals using a convolutional neural network
(CNN) [7]. Based on the PQ models in [7], the sampling frequency of signals is
set as 3200Hz, the fundamental frequency of signals is set as 50Hz, the number
of total cycles (periods) of the fundamental frequency is set as 10, and the
amplitude (per unit) is set as 1. Therefore, the dimension of input signal
vectors is 640. The network architecture is given in Fig. 2. In the PQ
assessment task, a CNN model was trained to classify signals into 17
categories. Signal types and corresponding representative symbols are shown in
Table II. We use the mathematical model of PQ disturbances and parameters
proposed in [3]. A labeled dataset of 255000 signals at 30dB of SNR (Signal-
Noise-Ratio) was constructed using 15000 signals belonging to each class. The
dataset was randomly shuffled, 1/4 of the shuffled data was split as the
testing set, and the rest was used for training.

TABLE II: Signal types and corresponding representative symbols.

C-1 | Normal | C-7 | Harmonics | C-13 | Sag with Oscillatory transient
---|---|---|---|---|---
C-2 | Sag | C-8 | Harmonics with Sag | C-14 | Swell with Oscillatory transient
C-3 | Swell | C-9 | Harmonics with Swell | C-15 | Sag with Harmonics
C-4 | Interruption | C-10 | Flicker | C-16 | Swell with Harmonics
C-5 | Transient/Impulse/Spike | C-11 | Flicker with Sag | C-17 | Notch
C-6 | Oscillatory transient | C-12 | Flicker with Swell |  |

Figure 2: Architecture of the CNN used in the experiments, where Conv(number
of features, kernel size, stride) denotes a convolution layer, MaxPool(kernel
size, stride) denotes a pooling layer and BN denotes batch normalization.

In order to analyze accuracy of a learning model using clean signals, we
randomly shuffled the dataset, and trained and tested the model 10 times. We
trained each model for 15 epochs and Adam [45] was used as the optimizer
($\alpha=0.001$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e^{-8}$). We
obtained $92.05\%\pm 0.11$ and $92.01\%\pm 0.11$ training and test accuracy,
respectively. One trained model with $98.46\%$ test accuracy is used as our
attack target model. For SSA, we generated corresponding adversarial signals
for all clean signals including training data and test data. For SAA, we used
a subset of training signals to generate universal perturbations.

Figure 3: Experimental analysis of classification of randomly chosen clean
signals (original) and the corresponding adversarial signals generated using
the FGSM (first row), SSA (second row) and SAA (third row) methods. In titles
of subfigures, C-$i$ (C-$j$) denotes a case where a signal belongs to class
C-$j$ while its class label is predicted as C-$i$. The x-axis denotes time (s)
and the y-axis denotes Voltage/Current (V/A p.u.).

###  IV-B Analysis of Signal-Specific Adversarial Signals

We compare performance of our proposed SSA and FGSM [23] for generating
signal-specific adversarial signals. We evaluate their performance using
average robustness [30] by

$\hat{\rho}_{\mathrm{adv}}=\frac{1}{\sigma({D})}\sum_{\boldsymbol{x}\in{D}}\frac{\|\hat{\boldsymbol{r}}(\boldsymbol{x})\|_{2}}{\|\boldsymbol{x}\|_{2}},$
(9)

where $\hat{\boldsymbol{r}}(\boldsymbol{x})$ is the perturbation vector
generated using an attack method (e.g., FGSM or SSA), and $\sigma(D)$ denotes
the size of the test set $D$. As shown in Fig. 3, magnitudes of the
adversarial perturbations generated by the SSA are very small and the
adversarial signals are imperceptible to human operators. The $\epsilon$
parameter of the FGSM [23] is set as 0.5 to achieve the overall 0.88
misclassification rate which is proportion of signals whose labels are changed
when perturbations are applied. According to the characteristics of FGSM
attack, the larger the parameter $\epsilon$, the larger the adversarial
perturbations, and the higher the corresponding misclassification rate.
Besides, the selection of the parameter $\epsilon$ has no effect on the
running time. We choose $\epsilon=0.5$ to achieve roughly the same
misclassification rate as the other two methods, so as to analyze the effects
of different attack methods. In Fig. 3, we observe that magnitudes of the
adversarial perturbations generated by the FGSM [23] are larger and more
perceptible than those generated by the SSA.

In Table I, we compare misclassification rate, average robustness and average
running time of FGSM [23] and SSA. Table I shows that the SSA achieves a
higher misclassification rate with smaller perturbation compared with the
FGSM. Besides, we notice that the average running time spent for adversarial
generation using the SSA is larger than using the FGSM. For example, the
average running time spent for generating one adversarial signal is 0.22
seconds for the SSA, while that for the FGSM is 0.012 seconds. SSA spent most
of the time for optimization of perturbation using Algorithm 1.

TABLE III: Misclassification rate on the test set. Note that although
universal perturbations are computed on small sets of signals, the
misclassification rate on test set is large.

Number of signals in training set (ratio) | 5000 (1.96$\%$) | 10000 (3.92$\%$) | 15000 (5.88$\%$) | 20000 (7.84$\%$)
---|---|---|---|---
Misclassification rate ($\%$) | 28 | 35 | 52 | 74

###  IV-C Analysis of Signal-Agnostic Adversarial Signals

Unlike FGSM and SSA, SAA computes signal-agnostic (universal) perturbations
using a training set of signals. We observed that the average time spent for
training SAA to achieve misclassification rate 0.74 is 30 minutes. Less than
10$\%$ of the total signals were used to generate the universal perturbation.
Table III shows that one can build relatively generalizable universal
perturbations with few signals. Besides, additional time is not spent for
optimization during testing, and the universal perturbation is applied to each
test signal in real-time ($1.6\times 10^{-6}$ sec.). Universal perturbations
and adversarial samples generated using $\xi=1$ are shown in Fig. 3 (the
average $l_{2}$ norm of a signal is 18). In the experiments, the
misclassification rate can reach to 0.74 and most natural signals are
misclassified with high probability.

Fig. 3 shows that voltage values of the universal perturbations are smaller
and more imperceptible to human operators compared to the perturbations
generated by FGSM. Although SSA and SAA can generate perceptually similar
adversarial samples for some classes, the samples generated by different
methods can be misclassified to different classes. For instance, class labels
of the adversarial samples belonging to $C$-3 which are generated by SSA and
SAA, are incorrectly predicted by the learned model as $C$-12 and $C$-17,
respectively.

###  IV-D Analysis of Performance of Black-Box Attacks

For training substitute models, we suppose that the attacker can access some
data in the training set (obtained by invading a database or building an
experimental platform for simulation). Two types of black-box setting are
considered:

  * •

Type I: The attacker knows the network architecture of the CNN model and does
not know the specific model parameters.

  * •

Type II: Both network architecture and specific parameters of the CNN model
are not known to attackers (the substitute model architecture has several
layers more or less than the corresponding target model in our experiments).

Besides, we also consider datasets of different sizes used for training
substitute models, that is, using $1/20$, $1/10$ and $1/5$ of the entire
training set. For each black-box type and training data ratio, we trained 10
substitute models independently. The average test accuracy values of well-
trained substitute models are shown in Table IV. Based on the well-trained
substitute models, we generated adversarial signals based on the proposed
attack methods in Section II. In detail, for each well-trained substitute
model, we generated corresponding adversarial signals against the entire test
data. Then, we use these generated adversarial signals to attack the target
model.

Since two attack methods (SSA and SAA) are proposed in Section II, we use
these two methods independently to generate adversarial signals and evaluate
their effects. The average misclassification rates of adversarial signals
against the target model are shown in Table V and VI, respectively. Table V
shows that black-box attacks for SSA have a low success rate. Besides, the
average misclassification rate of Type I is higher than that of Type II, which
implies that the substitute model using the same architecture with the target
model has a higher success rate. In contrast, Table VI implies that black box
attacks for SAA have a high success rate, and there is no significant
difference in the misclassification rate under the two black-box types. This
interesting phenomenon reflects that in black-box attacks, signal-agnostic
(universal) perturbations have a higher misclassification rate than signal-
specific perturbations. The above experimental results show that the attacker
can train his substitute models and generate corresponding universal
perturbations to carry out black-box attacks, which is a huge and practical
threat for CNN based PQ classification.

TABLE IV: Average test accuracies of well-trained substitute models.

Training data ratio | $1/20$ | $1/10$ | $1/5$
---|---|---|---
Type I | 94.9$\%$ | 96.4$\%$ | 99.3$\%$
Type II | 93.9$\%$ | 97.3$\%$ | 94.4$\%$

TABLE V: Average misclassification rate of black-box attacks for SSA.

Training data ratio | $1/20$ | $1/10$ | $1/5$
---|---|---|---
Type I | 4.7$\%$ | 5.1$\%$ | $5.3\%$
Type II | 3.5$\%$ | 3.6$\%$ | 4.1$\%$

TABLE VI: Average misclassification rate of black-box attacks for SAA.

Training data ratio | $1/20$ | $1/10$ | $1/5$
---|---|---|---
Type I | 73.2$\%$ | 74.1$\%$ | 75.6$\%$
Type II | 72.6$\%$ | 75.5$\%$ | 76.1$\%$

###  IV-E Graphical Analysis of Class Predictions under Attacks

We normalized confusion matrices generated according to class predictions
obtained using SAA, FGSM and SSA in Fig. 4, Fig. 5 and Fig. 6, respectively.
For comparison, the normalized confusion matrix based on the normal test
signals is also provided in Fig. 7. These normalized confusion matrices are
used to construct class-confusion graphs proposed below.

Figure 4: A normalized confusion matrix obtained using the SAA.

Figure 5: A normalized confusion matrix obtained using the FGSM.

Figure 6: A normalized confusion matrix obtained using the SSA.

Figure 7: A normalized confusion matrix obtained using the normal test
signals.

We analyze how class predictions of learned models change under attacks
exploring class confusion statistics. We construct a directed graph
${G=\left(V,E\right)}$, where each node $v_{i}\in V$ denotes a class label
C-$i$. A weighted directed edge $e_{i,j}\in E$ connecting a node $v_{i}$ to
the node $v_{j}$ indicates that signals belonging to the class ${\rm C-}i$,
are misclassified as ${\rm C-}j$. The weight $w_{i,j}$ of the edge $e_{i,j}$
is the ratio of the number of samples of the class ${\rm C-}i$ which are
misclassified as ${\rm C-}j$, to the total number of samples in the class
${\rm C-}i$.

((a)) SAA

((b)) FGSM

((c)) SSA

Figure 8: Visualization of graphs for each attack method. Directions of edges
are depicted by color instead of using directed arrows. For instance, the edge
$e_{10,17}$ is directed from the node $C$-10 to the node $C$-17. Therefore,
the color of the edge $e_{10,17}$ which is green, is same to that of the node
$C$-10 in (a). Thickness of edges is determined using normalized covariance
matrices. For instance, in (a), the number of samples belonging to $C$-10
which are misclassified as $C$-17 is larger than that of samples belonging to
$C$-16 which are misclassified as $C$-17. TABLE VII: Degrees of nodes of the
class-confusion graphs of FGSM, SSA and SAA.

FGSM | C-1 | C-2 | C-3 | C-4 | C-5 | C-6 | C-7 | C-8 | C-9 | C-10 | C-11 | C-12 | C-13 | C-14 | C-15 | C-16 | C-17
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
In-degree ($d_{in}$) | 0 | 0 | 0 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 3 | 11 | 0 | 0 | 16
Out-degree ($d_{out}$) | 2 | 2 | 2 | 3 | 2 | 2 | 2 | 3 | 2 | 2 | 2 | 2 | 2 | 2 | 3 | 2 | 3
SSA | C-1 | C-2 | C-3 | C-4 | C-5 | C-6 | C-7 | C-8 | C-9 | C-10 | C-11 | C-12 | C-13 | C-14 | C-15 | C-16 | C-17
In-degree ($d_{in}$) | 1 | 2 | 5 | 1 | 2 | 3 | 7 | 5 | 4 | 1 | 3 | 3 | 4 | 4 | 2 | 1 | 5
Out-degree ($d_{out}$) | 4 | 1 | 1 | 3 | 5 | 3 | 1 | 5 | 5 | 2 | 3 | 4 | 3 | 4 | 3 | 3 | 3
SAA | C-1 | C-2 | C-3 | C-4 | C-5 | C-6 | C-7 | C-8 | C-9 | C-10 | C-11 | C-12 | C-13 | C-14 | C-15 | C-16 | C-17
In-degree ($d_{in}$) | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 4 | 1 | 0 | 0 | 0 | 6 | 5 | 0 | 0 | 16
Out-degree ($d_{out}$) | 1 | 2 | 3 | 3 | 1 | 1 | 2 | 3 | 4 | 1 | 3 | 3 | 2 | 2 | 3 | 3 | 1

Class-confusion graphs computed for each attack method are visualized in Fig.
8. For the sake of clearance of visualizations, we only depict connections
denoting classification probability greater than chance555There are 17 types
of PQ signals and the number of signals of each type is the same. Therefore,
we assume that the probability of each signal being classified into any kind
of signal is $1/17$ (the assumption is only to better show corresponding
class-confusion graphs, because displaying large number of edges in figures
makes corresponding graphs look messy)., which is $1/17$. Distributions of
node degrees are given in Table VII. Out-degree value of a node $v_{i}$
associated with a class C$-i$ denotes the number classes to which samples of
the C$-i$ are incorrectly assigned by misclassification. For instance, samples
belonging to C$-15$ which are attacked by SSA are misclassified to the other
three different classes C$-2$, C$-8$ and C$-11$ (see Fig. 8). In-degree value
of a node $v_{i}$ denotes the number classes whose samples are misclassified
as C$-i$. For instance, samples belonging to C$-8$ and C$-13$ are
misclassified to C$-15$ (see Fig. 8).

Fig. 8(a) and Fig. 8(b) show that perturbations generated by the FGSM and SAA
make learned models misclassify signals to a few dominant classes such as
$C$-7, $C$-13$,C$-14 and $C$-17. We conjecture that signals belonging to these
dominant classes are distributed in larger areas in feature spaces (see Fig.
10), and thereby, models can be fooled by assigning labels of signals to these
classes with higher probability compared to the other classes. Besides, most
of the signals belonging to $C$-17, are not misclassified when universal
perturbations are applied. The reason may be that decision boundary of $C$-17
may be more complicated compared to that of the other classes, and located far
from that of the other classes (see Fig. 10). Therefore, universal
perturbations may not be able to fool models for misclassifying signals of
$C$-17.

In order to analyze the statistical relationship between classes whose samples
are successfully attacked and failed to be attacked, we compute entropy of
each node $v_{i}$ by

$H_{w}(v_{i})\triangleq-\sum_{j=1}^{d_{i}}{p_{i,j}\log(p_{i,j})},\;{\rm
and}\;p_{i,j}=\frac{w_{i,j}}{\sum\limits_{j=1}^{d_{i}}{w_{i,j}}},$ (10)

where $d_{i}\in\\{d_{in},d_{out}\\}$ is either in-degree or out-degree of
$v_{i}$. We compute the entropy for all the nodes of the graph $G$ by

$H_{w}(G)\triangleq\sum_{v_{i}\in V}H_{w}(v_{i}).$ (11)

We calculated unweighted graph entropy $H(G)$ using ${w_{i,j}=1},\forall
e_{i,j}\in E$. Entropy values given in Table VIII show that SSA provides the
largest entropy values. This result shows that signals misclassified under SSA
are more equally likely distributed to incorrect classes compared to FGSM and
SAA. We obtained the lowest values of $H_{w}(G)$ and $H(G)$ for the in-degree
distribution, and those of $H_{w}(G)$ for the out-degree distribution using
FGSM. However, SAA provided the lowest unweighted entropy for the out-degree
distribution. This result indicates that SAA makes the learned model
misclassify samples to less number of particular signal classes with higher
probability compared to FGSM and SAA. This observation is depicted for the
graph computed using the SAA with less number of node hubs in Fig. 8(a),
compared to the graphs computed using the FGSM (Fig. 8(b)) and the SSA (Fig.
8(c)).

TABLE VIII: Weighted and unweighted graph entropy values.

Entropy | Out-degree ($d_{out}$) | In-degree ($d_{in}$)
---|---|---
$H_{w}(G)$ | FGSM | SSA | SAA | FGSM | SSA | SAA
13.10 | 21.29 | 13.97 | 10.87 | 20.84 | 11.75
$H(G)$ | FGSM | SSA | SAA | FGSM | SSA | SAA
19.34 | 25.06 | 17.09 | 12.04 | 23.53 | 13.49

TABLE IX: Comparison of classification accuracy ($\%$) for adversarial
training (adv.) using SSA.

Adv. Training Epochs | Acc. on Training Data | Acc. on Test Data | Acc. on Adv. Training Data | Acc. on Adv. Test Data
---|---|---|---|---
0 | 98.54 | 98.46 | 2.09 | 2.09
1 | 91.31 $\pm$ 10.03 | 91.08 $\pm$ 11.33 | 90.14 $\pm$ 10.14 | 90.06 $\pm$ 10.12
10 | 83.26 $\pm$ 7.18 | 83.14 $\pm$ 8.29 | 93.15 $\pm$ 5.72 | 93.26 $\pm$ 5.52

###  IV-F Analysis of Performance of Adversarial Training

We fine-tuned the learned model for 10 additional epochs using generated
adversarial signals to improve its robustness against adversarial
perturbation. Although we generated adversarial signals for training and test
data, only adversarial signals in training data were used for adversarial
training. Adversarial signals in test data were used to evaluate performance
of adversarial training. For each trial, same adversarial signals were used in
all additional epochs. We fine-tuned the model for 10 more epochs using the
original data for a fair comparison.

Change of $\hat{\rho}_{\mathrm{adv}}$ for different adversarial training
strategies is depicted in Fig. 9. The robustness $\hat{\rho}_{\mathrm{adv}}$
is estimated using SSA, since it is the best performing method, while worst
performing SAA is not used in this experiment (see Table I). Fig. 9 shows that
adversarial training using adversarial signals generated by SSA significantly
increases robustness of the learned models to attacks. For example, robustness
of the learned models is improved by 200$\%$ after training by three epochs.
However, training the learned models with adversarial signals generated by
FGSM [23] can decrease the robustness of the learned models (note that we used
relatively large perturbations by FGSM $\epsilon=0.5$ due to its high
misclassification rate). The reason of this result may be that the
perturbations generated by the FGSM are larger than those generated by the
SSA. Adversarial training with extremely perturbed signals decreases
robustness of the learned models to adversarial perturbations [30].

###  IV-G Accuracy-Robustness Trade-off

Next, learned models are fine-tuned using combined signals (mixing of normal
signals and adversarial signals generated by SSA) to analyze change of their
accuracy (FGSM and SAA are not used since they need relatively large
perturbations to achieve high misclassification rate). Results show that
although adversarial training improved the classification accuracy using
adversarial PQ signals to $94.09\%$, the classification accuracy of the model
retrained using normal PQ signals is decreased to $81.34\%$ after training
with 10 additional epochs (see Table IX). The reason may be that fine-tuning
of models with generated adversarial signals can decrease the model’s ability
to express normal signals to a certain extent. This phenomenon also exists in
other fields such as the image domain [46, 47]. We propose the trade-off
between maintaining accuracy of classifiers on normal PQ signals and improving
their robustness to adversarial signals as an open challenge, which needs to
be better utilized and resolved in future research.

###  IV-H Visualization of Learned Feature Representations of PQ Signals under
Attacks

Visual inspection of attacks on PQ signals in the Smart Grid is not a well-
defined phenomenon, since their interpretation is not clear using measurement
devices. To address this problem, we propose an approach by first projecting
high dimensional signal and feature vectors of samples to low dimensional
spaces. Then, we visualize samples in projected spaces to explore their
discrimination among PQ classes.

Figure 9: Analysis of change of average robustness $\hat{\rho}_{\mathrm{adv}}$
of the learned models using SSA and FGSM.

Figure 10: Visualization of the PQ signals in (a) $\mathcal{X}_{n}$, (b)
$\mathcal{F}_{n}^{r}$, (c) $\mathcal{F}_{n}^{tr}$, (d) $\mathcal{X}_{ssa}$,
(e) $\mathcal{F}_{ssa}^{tr}$, (f) $\mathcal{F}_{ssa}^{adv}$, (g)
$\mathcal{X}_{saa}$, (h) $\mathcal{F}_{saa}^{tr}$, and (i)
$\mathcal{F}_{saa}^{adv}$, with NH and classification accuracy (Acc).

We used t-SNE [24] for projection of high dimensional vectors. We first
projected signal vectors residing in input signal spaces and feature vectors
obtained at the last hidden layer of CNN models into a 2D space $\mathcal{X}$
and a 2D feature space $\mathcal{F}$, respectively. Then, we depicted samples
on the projected spaces by points which are colored according to their class
labels. Projection quality is evaluated by neighborhood hit (NH), which
indicates how well samples belonging to classes are visually discriminated.
For a given $k$ (in our work, $k=5$), the NH for a point $x$ is the ratio of
its $k$-nearest neighbors that belong to the same category as $x$ [48]. The NH
for the entire projection is the average NH over its points. 1000 randomly
chosen normal PQ signals, adversarial signals generated by SSA and SAA are
visualized in Fig. 10. Fig. 10(a), (d) and (g) show poor discrimination of
normal samples, and adversarial signals generated by SSA and SAA in the
projected spaces $\mathcal{X}_{n}$, $\mathcal{X}_{ssa}$ and
$\mathcal{X}_{saa}$, respectively. This result shows difficulty of
discriminating samples using the input signals (normal signals or adversarial
signals). This qualitative result is also confirmed by their low NH values.

In Fig. 10(b), samples are depicted in a 2D feature space
$\mathcal{F}_{n}^{r}$ obtained before training of CNN models. This result
indicates that a CNN model with randomly initialized parameters has a poor
visual discrimination. It is reasonable to hypothesize that visual
discrimination would be improved after training of models. This is related to
the conjecture that CNNs can learn features useful for human perception of
classes [49]. To analyze this hypothesis, we first trained a CNN model and
extracted features of samples at the last hidden layer of the model. Then, we
projected features into a 2D feature space denoted by $\mathcal{F}_{n}^{tr}$.
Samples are depicted in the projected space $\mathcal{F}_{n}^{tr}$ in Fig.
10(c). Compared to Fig. 10(a) and (b), visual discrimination is significantly
improved in Fig. 10(c) and the corresponding NH is 87.78$\%$ and the test
accuracy is 98.9$\%$.

Fig. 10(e) shows that features of adversarial signals generated by SSA are not
visually better discriminative compared with Fig. 10(c). Therefore, the
adversarial attacks affect visual discrimination of the features. The test
accuracy for the 1000 randomly chosen adversarial signals generated by SSA is
only 2.7$\%$. However, features of the adversarial signals are visually
distinguishable to a certain degree, since the SSA can successfully fool top
layer softmax classifiers for misclassification of samples. Visual
discrimination of samples is improved in a projected space
$\mathcal{F}_{ssa}^{adv}$ of features learned after adversarial training of
the model (see Fig. 10(f)). This result visually verifies the efficiency of
adversarial training.

In Fig. 10(i), we visualize samples generated by SAA in a projected space of
features $\mathcal{F}_{saa}^{adv}$ learned after adversarial training of the
model. Although the NH is only 57.78$\%$, visual discrimination of adversarial
PQ signals is improved by adversarial training. This may suggest that
adversarial training of models using adversarial PQ signals generated by an
attack method can improve their robustness against adversarial PQ signals
generated by other attack methods. Results obtained using our proposed
approach are complementary to those obtained by visualization of raw signals
in Fig. 3. We believe that researchers can use the proposed approach for
qualitative analysis of class discrimination of normal and adversarial PQ
signals under attacks in the Smart Grid.

##  V Conclusion and Discussion

We proposed signal-specific adversarial attack (SSA) and signal-agnostic
adversarial attack (SAA) methods to attack learning models for classification
of power quality signals by generating adversarial signals in the Smart Grid.
We proposed and evaluated black-box attacks based on transferable
characteristics and the above two methods. We adopted adversarial training to
defend learned models against adversarial signals. Experimental results show
that our proposed SSA provides less perturbation compared to the state-of-the-
art FGSM which was adapted for power signal analysis, while the SAA can
generate the universal perturbation that can fool learned models. The attack
method based on the universal signal-agnostic algorithm has a higher transfer
rate of black-box attacks than the attack method based on the signal-specific
algorithm, which is a huge and practical threat for CNN based PQ
classification. In the analyses, the proposed adversarial training method
improved robustness of the learning models. In addition, we proposed an
approach for qualitative analysis of normal power quality signals and visual
inspection of adversarial attacks. We elucidated the mechanisms behind
robustness of models to adversarial attacks corresponding to the quantitative
results using the proposed visualization approach. We believe that the
proposed approaches will be useful for researchers and practitioners to
analyze power systems and lead new research directions. In the future work, we
plan to employ our approaches for a wider class of attack models (such as
gray-box and black-box attacks with higher misclassification rates) and
defense methods for data, network and device security in the Smart Grid.

## References

  * [1] J. Tian, T. Li, F. Shang, K. Cao, J. Li, and M. Ozay, “Adaptive normalized attacks for learning adversarial attacks and defenses in power systems,” in _2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_. IEEE, 2019, pp. 1–6.
  * [2] “IEEE recommended practice for monitoring electric power quality,” _IEEE Std 1159-2009 (Revision of IEEE Std 1159-1995)_ , pp. 1–94, June 2009.
  * [3] R. Igual, C. Medrano, F. J. Arcega, and G. Mantescu, “Integral mathematical model of power quality disturbances,” in _2018 18th International Conference on Harmonics and Quality of Power (ICHQP)_. IEEE, 2018, pp. 1–6.
  * [4] O. P. Mahela, A. G. Shaik, and N. Gupta, “A critical review of detection and classification of power quality events,” _Renewable and Sustainable Energy Reviews_ , vol. 41, pp. 495–505, 2015.
  * [5] S. Deokar and L. Waghmare, “Integrated DWT–FFT approach for detection and classification of power quality disturbances,” _International Journal of Electrical Power & Energy Systems_, vol. 61, pp. 594–605, 2014.
  * [6] M. Mishra, “Power quality disturbance detection and classification using signal processing and soft computing techniques: A comprehensive review,” _International Transactions on Electrical Energy Systems_ , p. e12008, 2018\.
  * [7] S. Wang and H. Chen, “A novel deep learning method for the classification of power quality disturbances using deep convolutional neural network,” _Applied Energy_ , vol. 235, pp. 1126–1140, 2019.
  * [8] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _ArXiv:1412.6572_ , 2014.
  * [9] H. Xu, Y. Ma, H. Liu, D. Deb, H. Liu, J. Tang, and A. Jain, “Adversarial attacks and defenses in images, graphs and text: A review,” _arXiv preprint arXiv:1909.08072_ , 2019.
  * [10] E. R. Balda, A. Behboodi, and R. Mathar, “Perturbation analysis of learning algorithms: Generation of adversarial examples from classification to regression,” _IEEE Transactions on Signal Processing_ , vol. 67, no. 23, pp. 6078–6091, 2019.
  * [11] X. Fei, N. Shah, N. Verba, K.-M. Chao, V. Sanchez-Anguix, J. Lewandowski, A. James, and Z. Usman, “CPS data streams analytics based on machine learning for cloud and fog computing: A survey,” _Future Generation Computer Systems_ , vol. 90, pp. 435–450, 2019.
  * [12] M. C. Bor, A. K. Marnerides, A. Molineux, S. Wattam, and U. Roedig, “Adversarial machine learning in smart energy systems,” in _ACM e-Energy_ , 2019, pp. 413–415.
  * [13] Y. Chen, Y. Tan, and D. Deka, “Is machine learning in power systems vulnerable?” in _2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_. IEEE, 2018, pp. 1–6.
  * [14] I. Niazazari and H. Livani, “Adversarial attacks on grid events classification: An adversarial machine learning approach,” _arXiv preprint arXiv:1911.08011_ , 2019.
  * [15] A. Venzke and S. Chatzivasileiadis, “Verification of neural network behaviour: Formal guarantees for power system applications,” _IEEE Transactions on Smart Grid_ , vol. 12, no. 1, pp. 383–397, 2021.
  * [16] A. Sayghe, J. Zhao, and C. Konstantinou, “Evasion attacks with adversarial deep learning against power system state estimation,” in _2020 IEEE Power Energy Society General Meeting (PESGM)_ , 2020, pp. 1–5.
  * [17] A. Sayghe, O. M. Anubi, and C. Konstantinou, “Adversarial examples on power systems state estimation,” in _2020 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT)_, Delft, Netherlands, 2020\.
  * [18] Y. Chen, Y. Tan, and B. Zhang, “Exploiting vulnerabilities of load forecasting through adversarial attacks,” in _Proceedings of the Tenth ACM International Conference on Future Energy Systems_. ACM, 2019, pp. 1–11.
  * [19] X. Zhou, Y. Li, C. A. Barreto, J. Li, P. Volgyesi, H. Neema, and X. Koutsoukos, “Evaluating resilience of grid load predictions under stealthy adversarial attacks,” in _2019 Resilience Week (RWS)_ , vol. 1. IEEE, 2019, pp. 206–212.
  * [20] Z. Tang, J. Jiao, P. Zhang, M. Yue, C. Chen, and J. Yan, “Enabling cyberattack-resilient load forecasting through adversarial machine learning,” in _2019 IEEE Power & Energy Society General Meeting (PESGM)_. IEEE, 2019, pp. 1–5.
  * [21] C. Barreto and X. Koutsoukos, “Design of load forecast systems resilient against cyber-attacks,” in _International Conference on Decision and Game Theory for Security_. Springer, 2019, pp. 1–20.
  * [22] T. Liu and T. Shu, “Adversarial false data injection attack against nonlinear AC state estimation with ANN in smart grid,” in _International Conference on Security and Privacy in Communication Systems_. Springer, 2019, pp. 365–379.
  * [23] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [24] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, no. 86, pp. 2579–2605, 2008\.
  * [25] B. D. Rouani, M. Samragh, T. Javidi, and F. Koushanfar, “Safe machine learning and defeating adversarial attacks,” _IEEE Security & Privacy_, vol. 17, no. 2, pp. 31–38, 2019.
  * [26] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [27] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2017, pp. 39–57.
  * [28] A. Modas, S.-M. Moosavi-Dezfooli, and P. Frossard, “Sparsefool: a few pixels make a big difference,” in _Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 9087–9096.
  * [29] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 5, pp. 828–841, 2019.
  * [30] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of 2016 IEEE conference on computer vision and pattern recognition (CVPR)_ , 2016, pp. 2574–2582.
  * [31] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” _arXiv preprint arXiv:1607.02533_ , 2016.
  * [32] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in _2016 IEEE European symposium on security and privacy (EuroS &P)_. IEEE, 2016, pp. 372–387.
  * [33] A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard, “The robustness of deep networks: A geometrical perspective,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 50–62, 2017.
  * [34] C. Liao, H. Zhong, A. Squicciarini, S. Zhu, and D. Miller, “Backdoor embedding in convolutional neural network models via invisible perturbation,” _arXiv preprint arXiv:1808.10307_ , 2018.
  * [35] M. Esmaeilpour, P. Cardinal, and A. L. Koerich, “From sound representation to model robustness,” _arXiv preprint arXiv:2007.13703_ , 2020.
  * [36] J. Lu, T. Issaranon, and D. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” in _Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 446–454.
  * [37] G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio, “On the number of linear regions of deep neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 2924–2932.
  * [38] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the expressive power of deep neural networks,” in _International conference on machine learning_. PMLR, 2017, pp. 2847–2854.
  * [39] E. Wong and Z. Kolter, “Provable defenses against adversarial examples via the convex outer adversarial polytope,” in _International Conference on Machine Learning (ICML)_. PMLR, 2018, pp. 5286–5295.
  * [40] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in _Proceedings of 2017 IEEE conference on computer vision and pattern recognition (CVPR)_ , 2017, pp. 1765–1773.
  * [41] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017, pp. 506–519.
  * [42] N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples,” _arXiv preprint arXiv:1605.07277_ , 2016.
  * [43] H. Ruitong, X. Bing, S. Dale, and S. Csaba, “Learning with a strong adversary,” _arXiv preprint arXiv:1511.03034_ , 2015.
  * [44] Y. Wu, D. Bamman, and S. Russell, “Adversarial training for relation extraction,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 1778–1783.
  * [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [46] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” _arXiv preprint arXiv:1901.08573_ , 2019.
  * [47] E. Arani, F. Sarfraz, and B. Zonooz, “Adversarial concurrent training: Optimizing robustness and accuracy trade-off of deep neural networks,” _arXiv preprint arXiv:2008.07015_ , 2020.
  * [48] P. E. Rauber, S. G. Fadel, A. X. Falcao, and A. C. Telea, “Visualizing the hidden activity of artificial neural networks,” _IEEE transactions on visualization and computer graphics_ , vol. 23, no. 1, pp. 101–110, 2016.
  * [49] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of 2018 IEEE conference on computer vision and pattern recognition (CVPR)_ , 2018, pp. 586–595.