# JVS corpus: free Japanese multi-speaker voice corpus

###### Abstract

Thanks to improvements in machine learning techniques, including deep
learning, speech synthesis is becoming a machine learning task. To accelerate
speech synthesis research, we are developing Japanese voice corpora reasonably
accessible from not only academic institutions but also commercial companies.
In 2017, we released the JSUT corpus, which contains 10 hours of reading-style
speech uttered by a single speaker, for end-to-end text-to-speech synthesis.
For more general use in speech synthesis research, e.g., voice conversion and
multi-speaker modeling, in this paper, we construct the JVS corpus, which
contains voice data of $100$ speakers in three styles (normal, whisper, and
falsetto). The corpus contains 30 hours of voice data including 22 hours of
parallel normal voices. This paper describes how we designed the corpus and
summarizes the specifications. The corpus is available at our project
page111https://sites.google.com/site/shinnosuketakamichi/research-
topics/jvs_corpus.

Index Terms: voice corpus, Japanese, speech synthesis

##  1 Introduction

Thanks to developments in deep learning techniques, studies on speech have
been targeted actively [1, 2, 3, 4]. Nowadays, speech synthesis, e.g., text-
to-speech, singing voice synthesis, voice conversion, and speech coding, is
becoming a machine learning task. Easily accessible voice corpora help to not
only accelerate speech-related research but also improve the reproductivity of
a study. In 2017, we released a large-scaled Japanese speech corpus, named the
JSUT corpus [5], for end-to-end text-to-speech synthesis. The corpus included
10 hours of reading-style speech data uttered by a single native Japanese
speaker and all pronunciations of daily-use characters and individual readings
in Japanese [6]. Since Oct. 2017, the project page [7] was accessed more than
6,000 times (75% from Japan and 25% from foreign countries) from more than 60
countries. We believe that the JSUT corpus has become one of the most used
Japanese corpora for modern speech synthesis research [8, 9].

Towards more general purposes of speech-related research, this paper
introduces a new Japanese voice corpus, named the JVS (Japanese versatile
speech) corpus. The corpus is designed to have many benefits for many types of
users as follows.

  * High-quality format: The audio files are sampled at 24 kHz, encoded at 16 bit, and formatted in RIFF WAV.

  * High-quality recording: The recordings were controlled by a professional sound director and done in a recording studio.

  * Many speakers: The corpus includes 100 native Japanese speakers, and all of the speakers are professional, e.g., voice actor/actress.

  * Many styles: Each speaker utters not only normal speech but also whisper and falsetto voices.

  * Large in scale: In total, the corpus contains 30 hours of voice data.

  * Parallel/non-parallel utterances: Each speaker utters parallel, i.e., common among speakers, and non-parallel, i.e., completely different among speakers, utterances.

  * Many tags: The corpus includes not only voice data but also transcriptions, gender information, $F_{0}$ ranges, speaker similarity, and phoneme alignments.

  * Free for research: The corpus is free to use for research in academic institutions and commercial companies.

  * Easily accessible: The corpus is freely downloadable online.

The next section describes how we designed the corpus.

##  2 Corpus design

The corpus consists of the following four sub-corpora. Their names are
formatted as [NAME][NUM_UTT]. [NUM_UTT] indicates the number of utterances per
speaker.

  * parallel100: 100 parallel normal (reading-style) utterances

  * nonpara30: 30 non-parallel normal utterances

  * whisper10: 10 whisper utterances

  * Falsetto10: 10 falsetto utterances

The directory structures of the corpus are listed below. The speaker name is
formatted as jvs[SPKR_ID]. [SPKR_ID] indicates the speaker ID with the range
of 1 through 100. .1  jvs001. .2  parallel100. .3  wav24kHz16bit. .3  lab. .3
transcripts_utf8.txt. .2  nonpara30. .3  wav24kHz16bit. .3  lab. .3
transcripts_utf8.txt. .2  whisper10. .3  wav24kHz16bit. .3
transcripts_utf8.txt. .2  falsetto10. .3  wav24kHz16bit. .3
transcripts_utf8.txt. .1  jvs002. .1  …. .1  jvs100. .1
speaker_similarity_male.csv. .1  speaker_similarity_female.csv. .1
duration.txt. .1  gender_f0range.txt.

###  2.1 Sub-corpora

This section describes how we designed the four sub-corpora.

####  2.1.1 parallel100

Parallel voices, i.e., utterances that are common among speakers, are used for
voice conversion [10, 11], speaker factorization [12], multi-speaker modeling
[8], and so on. We used 100 phonetically-balanced sentences of the sub-corpus
“voiceactress100”222The original sentences are included in the Voice Actress
Corpus [13], and the one included in the JSUT corpus had commas added at the
phrase break positions. of the JSUT corpus [5], and we let speakers utter the
sentences. This corpus contains not only the audio files but also the
transcriptions (stored in “parallel100/transcript_utf8.txt”) and phoneme
alignment (stored in “parallel100/lab”).

####  2.1.2 nonpara30

The use of non-parallel voices, i.e., utterances that are completely different
among speakers, is a challenging but more realistic situation than that of
parallel voices. Sentences to be uttered are randomly selected from the JSUT
corpus excluding its sub-corpus “voiceactress100.” Each speaker uttered 30
utterances that are different among speakers. This sub-corpus also includes
transcriptions and phoneme alignments. Note that, the sentences are not
phonetically balanced unlike the sub-corpora “parallel100.”

####  2.1.3 whisper10

Whispering is used to quietly communicate, i.e., convey secret information
without being overheard. Analysis [14], synthesis [15], recognition [16] and
conversion [17] of whispered voices have the potential to augment our silent-
speech communication. The first five sentences of this sub-corpus are the same
as those of the sub-corpus “parallel100,” and they are parallel among
speakers. The remaining five sentences are the same to those of the sub-corpus
“nonpara30,” and they are non-parallel among speakers. Namely, ten utterances
per speaker are parallel between whispered voices and normal voices.

####  2.1.4 Falsetto10

Falsetto is a vocal register occupying the $F_{0}$ range that is higher than
normal voices. The physiology of falsetto is different from that of normal
voices [18], and the analysis and synthesis of falsetto are remaining tasks
for signal processing-based vocoders. The first five sentences of this sub-
corpus are the same as those of the sub-corpus “parallel100.” The remaining
five sentences are the same as those of the sub-corpus “nonpara30” but
different to those of the sub-corpus “whisper10.” Namely, five utterances are
parallel among speakers, ten are parallel between normal voice and falsetto,
and five are parallel between whisper and falsetto.

###  2.2 Tags

This section describes some of the annotation results.

  * •

$F_{0}$ range (gender_f0range.txt): Typical pitch extractors, e.g., [19, 20,
21], have a range for $F_{0}$ search, and the setting is critical for the
results ultimately obtained for the voices. This corpus contains manually
annotated $F_{0}$ ranges per speaker for his/her normal voices.

  * •

Speaker similarity (speaker_similarity_*.csv): Perceptual similarity between
speakers is useful for selecting speakers (or models) [22] and modeling
speaker space [23]. This corpus contains perceptual similarity scores between
all pairs of speakers of each gender.

  * •

Duration (duration.txt): Duration, i.e., data size, and speech rate are also
included. Phoneme-level duration is calculated from the results of phoneme
alignments.

##  3 Results of data collection

Table 1: Speaker-wise duration statistics. Silence parts were included to calculate these values. | Mininum [min.] | Average [min.] | Maximum [min.] | Total (100 speaker) [hour]
---|---|---|---|---
parallel100 (100 utterances) | 10.11 (jvs020) | 13.11 | 18.24 (jvs084) | 22
nonpara30 (30 utterances) | 2.12 (jvs099) | 2.62 | 3.86 (jvs036) | 4.4
whisper10 (10 utterances) | 0.95 (jvs045) | 1.24 | 1.69 (jvs018) | 2.0
falsetto10 (10 utterances) | 0.90 (jvs045) | 1.18 | 1.61 (jvs035) | 2.0
Total | - | - | - | 30.4

###  3.1 Corpus specs

We hired 100 native Japanese professional speakers, which included 49 male and
51 female speakers. Their voices were recorded in a recording studio.
Recording for each speaker was done within one day. The recordings were
controlled by a professional sound director. The voices were originally
sampled at 48 kHz and downsampled to 24 kHz by SPTK [24]. The 16-bit/sample
RIFF WAV format was used. Sentences (transcriptions) were encoded in UTF-8.
The full context and monophone labels were automatically generated by Open
JTalk [25]. The phoneme alignments were automatically generated by Julius
[26]. $F_{0}$ ranges were manually annotated in accordance with hands-on voice
conversion [27]. The WORLD vocoder [20, 28] extracted $F_{0}$. Commas were
added between breath groups. For annotating perceptual similarity scores, we
followed Saito et al.’s study [23] and used a crowdsourcing service, Lancers
[29], which is a famous crowdsourcing service in Japan. Each listener scored
the perceptual similarity for each pair of speakers from $-3$ (completely
different) and $+3$ (very similar). A final score for each speaker pair was
obtained by averaging listeners’ scores. Ten different listeners scored each
speaker pair, and 1,000 listeners participated in total.

###  3.2 Analysis

####  3.2.1 Duration

Table 1 lists the statistics for speaker-wise duration. This corpus contains
26 hours of normal voices and 4 hours of other-style voices. Each speaker
uttered approximately 15.7 minutes of normal voices, 1.24 minutes of whispered
voices, and 1.18 minutes of falsetto. In the sub-corpus “parallel100,” the
transcription was common among speakers, but the duration was very different;
speaker “jvs084” uttered 1.8 times slower than speaker “jvs020.”

####  3.2.2 Perceptual speaker similarity

Fig. 1 shows matrices of perceptual similarity scores. For example, the most
similar pair was “jvs019” and “jvs096.” Also, a speaker that was most
dissimilar from the other speakers was “jvs010.”

##  4 Conclusion

In this paper, we constructed a corpus named the JVS corpus. The corpus was
designed for speech-related research using multi-speaker and multi-style
voices. Text data of the corpus is licensed as shown in the LICENCE file in
the JSUT corpus [7]. The tags are licensed with CC BY-SA 4.0. The audio data
may be used for

  * •

Research by academic institutions

  * •

Non-commercial research, including research conducted within commercial
organizations

  * •

Personal use, including blog posts.

Our project page at
https://sites.google.com/site/shinnosuketakamichi/research-topics/jvs_corpus
describes the terms for commercial use.

##  5 Acknowledgements

Part of this work was supported by the GAP foundation program of the
University of Tokyo and the MIC/SCOPE #182103104.

Figure 1: Speaker similarity matrix of 51 Japanese females and (b) its sub-
matrix obtained by large-scale subjective scoring.

## References

  * [1] G. Hinton, L. Deng, D. Yu, G. Dahl, A. r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury,  “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,”  Signal Processing Magazine of IEEE, vol. 29, no. 6, pp. 82–97, 2012\.
  * [2] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu,  “WaveNet: A generative model for raw audio,”  vol. abs/1609.03499, 2016.
  * [3] S. Takamichi, K. Tomoki, and H. Saruwatari,  “Sampling-based speech parameter generation using moment-matching network,”  in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017.
  * [4] Y. Saito, S. Takamichi, and H. Saruwatari,  “Statistical parametric speech synthesis incorporating generative adversarial networks,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 1, pp. 755–767, Jun. 2018.
  * [5] R. Sonobe, S. Takamichi, and H. Saruwatari,  “JSUT corpus: free large-scale japanese speech corpus for end-to-end speech synthesis,”  vol. arXiv abs/1711.00354, 2017.
  * [6] Governments of Japan Agency for Cultural Affairs,  “List of daily-use kanjis http://www.bunka.go.jp/kokugo_nihongo/sisaku/joho/joho/kijun/naikaku/kanji/index.html,”  2010\.
  * [7] “JSUT: Japanese speech corpus of Saruwatari Lab, the University of Tokyo corpus,” https://sites.google.com/site/shinnosuketakamichi/publication/jsut.
  * [8] S. Ueno, M. Mimura, S. Sakai, and Tatsuya Kawahara,  “Multi-speaker sequence-to-sequence speech synthesis for data augmentation in acoustic-to-word speech recognition,”  in Proc. ICASSP, Brighton, United Kingdom, May 2019, pp. 6161–6165.
  * [9] Z. Luo, J. Chen, T. Takiguchi, and Y. Ariki,  “Emotional voice conversion using dual supervised adversarial networks with continuous wavelet transform F0 features,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 10, pp. 1535–1548, Oct. 2019.
  * [10] T. Toda, A. W. Black, and K. Tokuda,  “Voice conversion based on maximum likelihood estimation of spectral parameter trajectory,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235, 2007.
  * [11] Y. Stylianou, O. Cappé, and E. Moulines,  “Continuous probabilistic transform for voice conversion,”  IEEE Transactions on Speech and Audio Processing, vol. 6, no. 2, pp. 131–142, Mar. 1998.
  * [12] H. Lu and S. King,  “Factorized context modeling for Text-to-Speech synthesis,”  in Proc. ICASSP. Vancouver, Canada, May 2013.
  * [13] y_benjo and MagnesiumRibbon,  “Voice-actress corpus,” http://voice-statistics.github.io/.
  * [14] T. Ito, K. Takeda, and F. Itakura,  “Analysis and recognition of whispered speech,”  Speech Communication, vol. 45, no. 2, pp. 139–152, 2005.
  * [15] V. A. Petrushin, L. I. Tsirulnik, and V. Makarova,  “Whispered speech prosody modeling for TTS synthesis,”  Chicago, U.S.A., May 2010.
  * [16] S.-C. Jou, T. Schultz, and A. Waibel,  “Whispery speech recognition using adapted articulatory features,”  in Proc. ICASSP, Philadelphia, U.S.A., Mar. 2005, vol. 1, pp. 1009–1012.
  * [17] T. Toda, M. Nakagiri, and K. Shikano,  “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 9, pp. 2505–2517, Nov. 2012.
  * [18] D. G. Childers and C. K. Lee,  “Vocal quality factors: Analysis, synthesis, and perception,”  The Journal of the Acoustical Society of America, vol. 90, no. 5, pp. 2394–2410, Nov. 1991.
  * [19] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne,  “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds,”  Speech Communication, vol. 27, no. 3–4, pp. 187–207, 1999.
  * [20] M. Morise, F. Yokomori, and K. Ozawa,  “WORLD: a vocoder-based high-quality speech synthesis system for real-time applications,”  IEICE transactions on information and systems, vol. E99-D, no. 7, pp. 1877–1884, 2016.
  * [21] D. Talkin,  “REAPER: Robust Epoch And Pitch EstimatoR,” https://github.com/google/REAPER.
  * [22] P. Lanchantin, Mark J.F. Gales, S. King, and J. Yamagishi,  “Multiple-average-voice-based speech synthesis,”  in Proc. ICASSP, Florence, Italy, May 2014, pp. 285–289.
  * [23] Y. Saito, S. Takamichi, and H. Saruwatari,  “DNN-based speaker embedding using subjective inter-speaker similarity for multi-speaker modeling in speech synthesis,”  in Proc. SSW10. Vienna, Austria, Sep. 2019.
  * [24] “Speech signal processing toolkit (SPTK),” http://sp-tk.sourceforge.net/.
  * [25] “Open jtalk,” http://open-jtalk.sourceforge.net/.
  * [26] A. Lee, T. Kawahara, and K. Shikano,  “Julius — an open source real-time large vocabulary recognition engine,”  in Proc. EUROSPEECH, Aalborg, Denmark, Sep. 2001, pp. 1691–1694.
  * [27] T. Toda,  “Hands on voice conversion,” https://www.slideshare.net/NU_I_TODALAB/hands-on-voice-conversion, 2018\.
  * [28] M. Morise,  “D4C, a band-aperiodicity estimator for high-quality speech synthesis,”  Speech Communication, vol. 84, pp. 57–65, 2016.
  * [29] “Lancers,” http://www.lancers.jp.


# Hard-label based Small Query Black-box Adversarial Attack

Jeonghwan Park, Paul Miller, Niall McLaughlin
Queen’s University Belfast, United Kingdom
{jpark04, p.miller<EMAIL_ADDRESS>
###### Abstract

We consider the hard-label based black-box adversarial attack setting which
solely observes the target model’s predicted class. Most of the attack methods
in this setting suffer from impractical number of queries required to achieve
a successful attack. One approach to tackle this drawback is utilising the
adversarial transferability between white-box surrogate models and black-box
target model. However, the majority of the methods adopting this approach are
soft-label based to take the full advantage of zeroth-order optimisation.
Unlike mainstream methods, we propose a new practical setting of hard-label
based attack with an optimisation process guided by a pre-trained surrogate
model. Experiments show the proposed method significantly improves the query
efficiency of the hard-label based black-box attack across various target
model architectures. We find the proposed method achieves approximately 5
times higher attack success rate compared to the benchmarks, especially at the
small query budgets as 100 and 250.

##  1 Introduction

Deep neural networks (DNNs) have become a most successful back-bone technique
adopted in many machine learning applications. Especially in the object
classification domain, DNN models can classify objects in images with near
human accuracy. However, such models are in general vulnerable to adversarial
attacks which use maliciously modified input examples to mislead a target
model. [29].

Depending on the amount or type of information of target models the attack
methods can access, adversarial attack methods are divided into two
categories: white-box and black-box attacks. White-box attacks are carried out
with full access to necessary information of the target models, such as their
weights and structures, to efficiently conduct back-propagation and calculate
gradients [17, 14]. In the black-box attack settings, however, the attack
methods can only monitor the input examples and corresponding output
information from the target model [8, 5, 20]. The black-box adversarial
attacks are, therefore, considered to be more practical attack methods
compared to the white-box attacks.

Carlini et al. [7] has emphasised the importancy of black-box adversarial
attack to evaluate the robustness or security of applications running with DNN
model. The applicable black-box attack method was introduced by Papernot et
al. [28] in early 2016. Black-box attacks are carried out without knowledge of
the target model, and rely on the target model’s output data obtained through
model querying. Therefore, such attack methods utilise optimisation processes
to approximate gradients to generate adversarial examples. Depending on the
method to approximate gradient, Black-box attack methods are broadly
categorised into transfer based and query based attacks. Transfer based
attacks utilise surrogate models to generate gradients which are directly used
as the approximate gradient, since such gradients are likely remain
adversarial for the target model due to their transferability [28, 27].
Although various methods have been introduced to improve the transferability
[14, 32], the attack success rate (ASR) is yet to be satisfactory. The reason
is that there lacks an adjustment procedure in transfer based attacks when the
gradient of the surrogate model points to a non-adversarial region of the
target model[13]. Query based attacks adopt various gradient-free optimisation
processes, which search the optimal point of the functions without using their
derivative, to estimate gradients [9, 31, 20]. Consider a DNN model which
classifies an input example into a class. The real-world applications, which
deal with the classification problem utilising the DNN, generate a predicted
soft-label and/or hard-label as the classification result. According to
Galstyan et al. [16], a hard label is one assigned to a member of a class
where membership is binary, and a soft label is one which has a score
(probability or likelihood) attached to it. Based on this definition, the
query based attacks subdivided further into soft-label based, and hard-label
based attacks. As the target model’s classification probability distribution
is not observable to the attacks, hard-label based attacks are generally
considered to be more challenging task compared with soft-label based attacks.
Although query based attack methods generally achieve a higher ASR in
comparison with the transfer based attacks, regardless of their categories
they typically suffer from a large number of queries required to perform a
successful attack.

In this report a novel method is introduced to tackle the query in-efficiency
of the hard-label based black-box attack methods by proposing a black-box
attack method called Small-Query Black-Box Attack (SQBA). SQBA integrates the
transfer based attack to take advantage of the gradients generated from a
surrogate model, and applies gradient-free optimisation introduced in [8]. In
summary, the contributions of this report are:

  * •

A novel transfer based iterative gradient estimation is proposed to guide
gradient direction in the black-box attack settings.

  * •

We design a hard-label based black-box attack method, SQBA, which has an
optimisation process guided by the proposed transfer based gradient
estimation.

  * •

Through the experiments the improved query-efficiency of SQBA attack method is
demonstrated in comparison with several state-of-the-art hard-label based
black-box attack methods.

##  2 Related works

In the black-box adversarial attack settings, the attack methods need to
generate adversarial examples without gradient information from the target
model, as they have no or limited accessibility to the model. A common choice
to tackle this problem is to utilise an approximate gradient instead for
generating adversarial examples. While the mainstream approach in generating
the approximate gradients is to numerically estimate them by the zeroth-order
optimisation algorithms[20, 8, 10, 5, 21], transfer based attacks take an
advantage of white-box attack method (surrogate model) to estimate the
gradients. Intuition in the transfer based attack methods is that adversarial
examples which successfully attack a surrogate model are likely to remain
adversarial for the target model due to their transferable characteristics
[27].

Tramer et. al. demonstrated in their work [30] that adversarial examples
display a high level of transferability between models, which are trained for
the same task, but have different architectures. To exploit this
characteristic of the adversarial examples, transfer based attack methods
train substitute (surrogate) models to achieve reasonably similar behaviour to
the target model, and then apply generic white-box attacks to produce
approximate gradients to attack the target model. In this report, however, it
is assumed a surrogate model which well clones the target model is given,
considering an easy access to the off the shelf pre-trained models, whose
network structures are different from the target model.

Adversarial examples generated from the surrogate model are not always
transferable to the target model, although many efforts have been made to
improve the transferability of the adversarial examples by utilising momentum
guidances [14, 32], or transforming input examples [33, 15, 25]. The failure
of transferability is often presented when the classification probabilities of
the models are different, and such discrepancy tends to cause disagreement on
gradient directions.

The query based attack methods commonly utilise the iterative-querying
approach adopting the zeroth-order optimisation. In details, the approximate
gradient is estimated by finite difference [9, 4], random-gradient estimation
[31] and natural evolution strategy (NES) [20, 21]. However, such attack
methods suffer from impractically large numbers of queries to achieve
successful attack. It is because they typically spend many queries to find the
intermediate adversarial examples as the nature of zeroth order optimisation
becomes effective if and only if examples lie near the decision boundary of
the target model [5, 10, 8].

To overcome this drawback of the query based attack methods, an approach to
leverage transfer based and query based methods have been adopted in recent
studies. Brunner et al. [6] integrated a white-box attack method within
Boundary Attack (BA) method [5], and used adversarial example originated from
the gradient of the surrogate white-box model to guide optimisation process.
Yang et al. [34] improved SimBA [18] query based attack and applied on-line
surrogate model updating algorithm in which simulates both forward and
backward process of target model. Zhou et al. [35] systematically integrated
the surrogate model to generate a Jacobian matrix and improved query
efficiency adopting optimisation process of SimBA. Dong et al. [13] obtained
the gradient information from a surrogate model, and used it to accelerate
searching process. Huang et al. [19] generated gradients from a surrogate
model, and applied NES [20] optimisation process to search adversarial
examples.

Most of methods reviewed here utilise the optimisation processes rely on the
soft-label output of the target model, whereas [6] focuses on the similar
attack settings as SQBA method which uses the hard-label optimisation process.

##  3 Problem Definition

Suppose $F(x)$ is a model to deal with a $k$-class object classification
problem, and the output of the model is denoted as
$f(x)\rightarrow\mathbb{R}^{k}$ to quantify classification probabilities of
the input example $x\in\mathbb{R}^{m}$ belonging to the classes. The
successful classification is represented as
$f_{c^{\dagger}}(x)=\max_{k}[f_{k}(x)]$, where $c^{\dagger}$ is the true class
of $x$.

The goal of the adversarial attack is to find a modified example $\tilde{x}$
such that $f_{\tilde{c}}(\tilde{x})>f_{c^{\dagger}}(\tilde{x})$ and
$\mathcal{D}(x,\tilde{x})$ is small enough, where $\tilde{c}$ is an
adversarial class and $\mathcal{D}$ is a dissimilarity metric. In hard-label
based black-box attack settings, the attack methods iteratively inquire the
target model with queries using purposely modified examples $x^{\prime}_{t}$
and only observe the predicted classes. The parameters $\theta$ in the target
model and the classification probabilities
$\\{f_{0}(x^{\prime}_{t}),f_{1}(x^{\prime}_{t}),...,f_{k}(x^{\prime}_{t})\\}$
are not accessible to the attack methods.

##  4 Optimisation Framework

The output vector of model $F(x)$ is a probability distribution over the class
set $\textbf{k}=\\{1,...,k\\}$. The classification function of $F(x)$ is
denoted as $C:\mathbb{R}^{m}\rightarrow\textbf{k}$ which maps input example
$x$ to the class with highest probability, and it is defined as [8]:

$C(x):=\operatorname*{arg\,max}\limits_{c\in\textbf{k}}[f_{c}(x)]$ (1)

Suppose a function $\mathcal{H}(x)$ is defined for the untargeted attack to
divert the true classification $c^{\dagger}=C(x)$ to any unknown class, which
is an adversarial class, $\tilde{c}\in\textbf{k}$ subject to $\tilde{c}\neq
c^{\dagger}$ as:

$\mathcal{H}(x+\alpha\mu):=\lim_{\alpha\rightarrow\epsilon}\Bigl{(}f_{\tilde{c}}(x+\alpha\mu)-\max\limits_{\tilde{c}\neq
c}[f_{c}(x+\alpha\mu)]\Bigr{)}$ (2)

where $\mu$ is a perturbation vector generated with an intention to conduct
adversarial attack, $\alpha$ is a scaling factor, and $\epsilon\leq 1$ is a
small positive value. For the targeted attack, the adversarial class
$\tilde{c}$ becomes a designated class. When $\mathcal{H}(x^{\prime})$ is
positive, given modified example $x^{\prime}$ is clearly adversarial, while
negative $\mathcal{H}(x^{\prime})$ means $x^{\prime}$ lies in the true class
region. Especially when $\mathcal{H}(x^{\prime})$ is zero, $x^{\prime}$ is
indicated to lie exactly on the decision boundary between two classes,
$c^{\dagger}$ and $\tilde{c}$. The examples which fall into this case are
called boundary-example [24].

In hard-label based attack setting, however, $\mathcal{H}(x^{\prime})$ is not
a linear function and observed as the Heaviside step function since $f_{c}(x)$
is not accessible. Therefore, $\mathcal{H}(x^{\prime})$ needs to be re-defined
as:

$\mathcal{H}(x^{\prime})\
:=\left.\begin{cases}\;\;\,1,\;\;\text{if}\;f_{\tilde{c}}(x^{\prime})>\max\limits_{\tilde{c}\neq
c}[f_{c}(x^{\prime})]\\\ -1,\;\;\text{otherwise.}\end{cases}\right.$ (3)

where $x^{\prime}=x+\alpha\mu$. When $\alpha$ is acceptably small, the status
change of $\mathcal{H}(x^{\prime})$ is observed if and only if $x^{\prime}$ is
a direct neighbour of the boundary-example. In the hard-label based attack
settings, therefore, it is necessary to keep the distance between
$x^{\prime}_{t}$ and decision boundary close enough in the search phase to
guarantee the successful attack with high quality adversarial examples.

As shown in Equation 3, $\mathcal{H}(x^{\prime})$ is a boolean function, and
therefore it is useful to indicate the successful adversarial examples. With
the boolean function, the objective of the adversarial attacks can be seen as
a process to find an example which satisfies:

$\min\limits_{x^{\prime}}\mathcal{D}(x^{\prime},x),\quad\text{such
that}\;\mathcal{H}(x^{\prime})=1$ (4)

where $\mathcal{D}$ is a distance metric to quantify the amount of applied
perturbation vector to modify the given example. Typically $l_{p}$-norm, such
that $p\in\\{0,2,\infty\\}$, is utilised for this purpose. In this work,
$l_{2}$-norm distance metric,
$\mathcal{D}(x^{\prime},x)=\|x^{\prime}-x\|_{2}$, is chosen.

Consider the objective of the adversarial attacks defined in Equation 4 as an
optimisation problem. Two gradients are estimated in this work to solve such
problem, and to effectively control the error in estimations. The first
gradient $\nabla\mathcal{H}_{w}(x^{\prime})$ is estimated via following
process:

$\nabla\mathcal{H}_{w}(x^{\prime}_{t}):=\mu_{t}^{i}\quad\text{subject
to}\;\min\limits_{\mu_{t}^{i}}\mathcal{D}(x,x^{\prime}_{t}+\delta_{t}\mu_{t}^{i})$
(5)

where $\mu_{t}^{i}=\nabla\mathcal{J}_{S}(x+v_{t}^{i},\;c^{\dagger})$ is a
gradient generated from a pre-trained surrogate model $S(x)$, which well
clones the classification behaviour of the target model $F(x)$, $v_{t}^{i}$ is
a vector to differentiate input example $x$, and $\delta_{t}$ is a scaling
factor for the iteration $t$. The gradient $\mu_{t}^{i}$ is calculated through
a white-box attack by injecting modified example $(x+v_{t}^{i})$ and true
class $c^{\dagger}$ of $x$ into the surrogate model $S(x)$. Specifically,
$l_{\infty}$ gradient from Dual Gradient Method (DGM), which is detailed in
the Supplemental Material, is used for this purpose. In each iteration $n$
gradients are generated as, $\\{\mu_{t}^{0},\mu_{t}^{1},...,\mu_{t}^{n}\\}$,
and a gradient which can move current intermediate adversarial example
$x^{\prime}_{t}$ to the closest point to the input example $x$ preserving
adversariality, is selected. This process is detailed further in section
Section 4.1 of this report. Boolean function $\mathcal{H}(x^{\prime})$
provides an approximation of the gradient direction, which has been utilised
in various black-box attack methods [8, 10, 20, 18]. However, Equation 5
estimates $\nabla\mathcal{H}_{w}(x^{\prime}_{t})$ including its likely
direction. Therefore, the proposed attack method doesn’t apply such
approximation to $\nabla\mathcal{H}_{w}(x^{\prime}_{t})$ for more effective
attack process.

Algorithm 1 SQBA Adversarial Example Computation

input: classifier f, example x, true class  $c^{\dagger},$

surrogate mode S

output: adversarial example  $\tilde{x}$

$t=0,\;\beta=1,\;x^{\prime}_{t}=\text{sign}\left(\nabla_{S}\mathcal{J}(x,c^{\dagger})\right)$

$x^{\prime}_{t+1}=\;$ Equation 9 $\leftarrow x,\;x^{\prime}_{t}$

while queries $\leq$ query budget do

$t=t+1,\;\delta_{t}=10^{-2}\mathcal{D}(x,x^{\prime}_{t})$

$\nabla\mathcal{H}_{w}=\;$ Equation 5 $\leftarrow\delta_{t}$

$\nabla\mathcal{H}_{b}=\;$ Equation 6 $\leftarrow\delta_{t}$

$g_{t}=\;$ Equation 7
$\leftarrow\nabla\mathcal{H}_{w},\nabla\mathcal{H}_{b},\beta$

$\dot{x}_{t},\;\beta=\;$ Equation 8 $\leftarrow x^{\prime}_{t},\;g_{t}$

$x^{\prime}_{t+1}=\;$ Equation 9 $\leftarrow x,\;\dot{x}_{t}$

end while

$\tilde{x}=x^{\prime}_{t}$

The gradient estimation algorithm used in HSJA attack method [8] is adopted in
the proposed attack method to estimate the second gradient
$\nabla\mathcal{H}_{b}(x^{\prime}_{t})$. The algorithm generates $p_{t}$
random examples $\mu_{t}^{i}$ which are uniformly distributed, where
$\mu_{t}^{i}\in\mathbb{R}^{m}$ and $i=\\{0,1,...,p_{t}\\}$. At every
iteration, the number of random examples is re-calculated. In the proposed
method smaller $p_{t}=10\sqrt{t+1}$, compared with HSJA, is sufficient as the
optimisation has already been progressed with
$\nabla\mathcal{H}_{w}(x^{\prime}_{t})$. The second gradient is then estimated
from the generated random examples via the Monte Carlo method as:

$\nabla\mathcal{H}_{b}(x^{\prime}_{t}):=\frac{1}{p_{t}}\sum^{p_{t}}_{i=1}\mathcal{H}(x^{\prime}_{t}+\delta_{t}\mu^{i}_{t})\mu^{i}_{t}$
(6)

where $\mu^{i}_{t}\sim\mathcal{N}(O,I)$. $\delta_{t}$ is a scaling factor
which is also applied in Equation 5, and updated at every iteration as
$\delta_{t}=10^{-2}\mathcal{D}(x,x^{\prime}_{t})$. With the estimated
gradients $\nabla\mathcal{H}_{w}(x^{\prime}_{t})$ and
$\nabla\mathcal{H}_{b}(x^{\prime}_{t})$, the iterate gradient $g_{t}$ is
approximated as:

$g_{t}\approx\beta_{t}\frac{\nabla\mathcal{H}_{w}(x^{\prime}_{t})}{\|\nabla\mathcal{H}_{w}(x^{\prime}_{t})\|_{2}}+(1-\beta_{t})\frac{\nabla\mathcal{H}_{b}(x^{\prime}_{t})}{\|\nabla\mathcal{H}_{b}(x^{\prime}_{t})\|_{2}}$
(7)

Gradient combining factor $\beta$ is obtained from a boolean function whose
output is switched between $\\{0,1\\}$. The proposed method begins with
$\beta=1$ meaning $\nabla\mathcal{H}_{w}(x^{\prime}_{t})$ is solely used to
approximate iterate gradients. When $\nabla\mathcal{H}_{w}(x^{\prime}_{t})$ is
indicated to be in a local minima, $\beta$ is then switched to 0 to start
using $\nabla\mathcal{H}_{b}(x^{\prime}_{t})$ instead.

Figure 1: Illustration of the process to find iterate example search
direction.

(a) CIFAR-10

(b) ImageNet

Figure 2: Change of Angle of the gradient vectors from surrogate model.

Consider an iterative optimisation process which is given access to the
approximate gradient $g_{t}$ with an intermediate adversarial example
$x^{\prime}_{t}$. The process performs an update as
$\dot{x}_{t}=x^{\prime}_{t}+\alpha g_{t}$. The optimisation problem in
Equation 4 becomes meaningful when $\dot{x}_{t}$ is a close neighbour to the
boundary-example. Therefore, a tuning process is necessary to move
$\dot{x}_{t}$ towards the boundary such as:

$\dot{x}_{t}=\min\limits_{\alpha}\left(x^{\prime}_{t}+\alpha
g_{t}\right),\quad\text{such that}\;\mathcal{H}(\dot{x}_{t})=1$ (8)

where $\alpha$ is a line search parameter. The tuning process shall guarantee
obtained $\dot{x}_{t}$ to satisfy $\mathcal{H}(\dot{x}_{t})=1$ and also
$\min\mathcal{D}(x^{\prime}_{t},\dot{x}_{t})$, so that the next iterate
example lies on the boundary. It can be noticed when an approximate gradient
$g_{t}$ gets into a local or global minima, the line search parameter $\alpha$
likely becomes small to find $\dot{x}_{t}$. Therefore, $\alpha$ is used to
switch the boolean function output $\beta$ in Equation 7. In practice
$\alpha\leq 1.0$ is used for this purpose.

The final step of the iterative optimisation is increasing the correlation
between input example $x$ and calculated intermediate adversarial example
$\dot{x}_{t}$. A binary search algorithm is applied for the improved
correlation as

$x^{\prime}_{t+1}=\lim\limits_{\zeta\rightarrow 0}\left(\zeta
x+(1-\zeta)\dot{x}_{t}\right),\;\text{such
that}\;\mathcal{H}(x^{\prime}_{t+1})=1$ (9)

The process shown in Equation 9 is a simple and greedy search algorithm,
however it is useful enough to be adopted in many hard-label based methods [8,
10].

###  4.1 Gradient from surrogate model

The transferability of gradients generated from the surrogate model is
mitigated when the decision boundary of the surrogate model does not closely
match the target model. However, even when this is the case, the decision
boundaries of both models may still be reasonably nearby [6]. To address this
drawback, the proposed SQBA attack utilises the multi-gradient method. Let
$\tilde{x}$ be an adversarial example generated from surrogate model $S(x)$,
and $\tilde{v}=\nabla_{S}\mathcal{J}(x,c^{\dagger})$ be a perturbation vector
to transpose the input example $x$ to $\tilde{x}$. The examples that lie on
$\tilde{v}$ are calculated by $x^{\prime}_{i}=x+\eta_{i}\tilde{v}$, where
$\eta_{i}\leq 1$ is a positive scaling factor which configures the distance
from input example, such as $x^{\prime}_{i}=x$ subject to $\eta=0$, and
$x^{\prime}_{i}=\tilde{x}$ subject to $\eta_{i}=1$. As illustrated in Figure
1, a set of gradients then can be generated from $S(x^{\prime}_{i})$ as:

$\mu_{i}=\nabla_{S}\mathcal{J}(x^{\prime}_{i},c^{\dagger}),\quad\text{where}\;i=0,1,...,n$
(10)

Dataset |  Model |  Accuracy
---|---|---
CIFAR-10 |  ResNet-18 |  93.07%
|  MobileNet-V2 |  93.90%
|  Inception-V3 |  93.74%
|  GoogLeNet |  92.96%
ImageNet |  EfficientNet-b3 |  83.58%
|  VGG-16 |  78.98%
|  ResNet-18 |  76.74%

Table 1: List of DNN Models and Classification Accuracies

Dataset |  Target Model |  Surrogate Model
---|---|---
CIFAR-10 |  ResNet-18 |  MobileNet-V2
|  |  GoogLeNet
|  MobileNet-V2 |  Inception-V3
|  |  ResNet-18
ImageNet |  EfficientNet-b3 |  ResNet-18
|  |  VGG-16
|  VGG-16 |  EfficientNet-b3
|  |  ResNet-18

Table 2: Target and Surrogate models mapping

Many works have proven that the optimal search direction in hard-label based
attack is perpendicular to $\tilde{v}$ [5, 8]. Suppose a function
$\mathcal{A}(\bar{a}_{1},\bar{a}_{2})$ is defined to derive the angle between
two vectors as
$\mathcal{A}(\bar{a}_{1},\bar{a}_{2}):=\cos\angle(\bar{a}_{1},\bar{a}_{2})$,
where
$\cos\angle(\bar{a}_{1},\bar{a}_{2})=(\bar{a}_{1}\cdot\bar{a}_{2})/\|\bar{a}_{1}\|_{2}\|\bar{a}_{2}\|_{2}$.
The angle between $\tilde{v}$ and a gradient vector $\mu_{i}$ is then
calculated with $\mathcal{A}(\tilde{v},\mu_{i})$.

To exam the direction of gradients of the set of examples
$x^{\prime}_{i},\;i=0,1,...,n$, dedicated 500 examples to each CIFAR-10 and
ImageNet-100 datasets were taken, and average direction of the gradients from
various DNN models were plotted in Figure 2. As one can expect, when
$\eta_{i}$ is small, in other words, $x^{\prime}_{i}$ is close to input
example $x$, the direction of gradient vector $\mu_{i}$ is near parallel to
$\tilde{v}$ as $\mathcal{A}(\tilde{v},\mu_{i})\approx 1$. The direction of
$\mu_{i}$, however, tends to be exponentially changed even at a small
increment in $\eta$, and rapidly saturated to the perpendicular direction to
$\tilde{v}$ as $\mathcal{A}(\tilde{v},\mu_{i})\approx 0$. Regardless of DNN
models or dataset, the similar pattern of changes is observed that is an
example which lies at $\eta_{i}\approx 0.2$, and afterwards their gradients
are almost saturated to $\mathcal{A}(\tilde{v},\mu_{i})\approx 0$. The iterate
approximate gradient, therefore, can be chosen from the calculated gradient
vectors such that $\eta_{i}\geq 0.2$ and
$\mathcal{H}(\tilde{x}+2\delta\mu_{i})=1$ as:

$\nabla\mathcal{H}_{w}(\tilde{x})=\mu_{i}\quad\text{where}\;i=\operatorname*{arg\,min}\limits_{\mathcal{D}}[\mathcal{D}(x,\tilde{x}+2\delta_{t}\mu_{i})]$
(11)

CIFAR-10: ResNet-18 at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  100 |  250 |  500 |  750 |  1000
Attack Methods |  HSJA [8] |  1.2% |  9.7% |  33.2% |  46.9% |  60.5%
BA [5] |  2.9% |  3.7% |  4.0% |  4.2% |  4.5%
Sign-Opt [10] |  8.5% |  10.5% |  11.4% |  24.5% |  32.1%
Biased-BA(GoogLeNet) [6] |  3.0% |  5.5% |  13.7% |  19.2% |  30.3%
SQBA(MobileNet-V2) |  38.4% |  52.1% |  66.1% |  74.7% |  82.0%
|  SQBA(GoogLeNet) |  28.1% |  40.4% |  60.5% |  73.0% |  79.2%

Table 3: Query Budgets and ASR of hard-label based black-box attacks on
ResNet-18 trained on CIFAR-10

CIFAR-10: MobileNet-V2 at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  100 |  250 |  500 |  750 |  1000
Attack Methods |  HSJA [8] |  3.9% |  23.4% |  67.6% |  78.4% |  85.6%
BA [5] |  9.3% |  10.0% |  11.9% |  13.1% |  16.1%
Sign-Opt [10] |  26.8% |  32.0% |  37.3% |  61.3% |  68.9%
Biased-BA(Inception-V3) [6] |  11.9% |  28.0% |  56.3% |  68.0% |  73.8%
SQBA(Inception-V3) |  71.9% |  80.2% |  89.4% |  91.9% |  95.6%
|  SQBA(ResNet-18) |  72.0% |  83.6% |  91.1% |  95.6% |  97.6%

Table 4: Query Budgets and ASR of hard-label based black-box attacks on
MobileNet-V2 trained on CIFAR-10

##  5 Experiments

Experimental analysis carried out to evaluate SQBA attack method is detailed
here. The efficiency of SQBA attack was compared with several state-of-the-art
attack methods which utilise the hard-label based adversarial attack setting.

###  5.1 Benchmarks

SQBA attack method was compared with four state-of-the-art black-box attack
methods: HSJA Attack [8], Sign-Opt Attack [10], BA Attack [5] and Biased-BA
Attack [6]. These attack methods commonly fall into the hard-label based
black-box adversarial attack setting as they solely observe the predicted
class against the given example. Especially Biased-BA is taking the similar
approach as SQBA which embeds a transfer based algorithm within the base line
hard-label based attack method. Model parameters or model output logits are
not accessible to the attack methods used in the experiments. All the
benchmark methods were configured to perform the untargeted attack scenario.
In the untargeted attack scenario, an attack becomes successful if the target
model predicts a different class from the true class of the given example.
Note that Biased-BA attack was initialised with a random noise instead of a
target example to conduct such an attack scenario. Attack strategies of the
benchmark methods commonly have three stages: First, they find an initial
adversarial example (starting point); Second, they iteratively search a path
towards a further optimised example, which is closer to the original example
but still maintains the adversariality; and finally, they stop searching when
the optimisation process satisfies stopping criteria which is given the query
budget.

###  5.2 Datasets and Models

ImageNet: EfficientNet-b3 at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  100 |  250 |  500 |  750 |  1000
Attack Methods |  HSJA [8] |  1.7% |  5.2% |  12.0% |  19.3% |  24.6%
BA [5] |  2.7% |  3.0% |  3.0% |  3.2% |  3.4%
Sign-Opt [10] |  5.5% |  7.5% |  7.2% |  7.5% |  7.3%
Biased-BA(VGG-16) [6] |  2.0% |  3.4% |  7.5% |  15.6% |  21.6%
SQBA(ResNet-18) |  33.8% |  38.2% |  46.8% |  49.4% |  55.2%
|  SQBA(VGG-16) |  30.4% |  37.4% |  45.0% |  48.7% |  53.2%

Table 5: Query Budgets and ASR of hard-label based black-box attacks on
EfficientNet-b3 trained on ImageNet

ImageNet: VGG-16 at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  100 |  250 |  500 |  750 |  1000
Attack Methods |  HSJA [8] |  2.2% |  10.2% |  24.0% |  31.2% |  40.8%
BA [5] |  5.0% |  6.0% |  6.8% |  6.2% |  9.0%
Sign-Opt [10] |  11.4% |  13.6% |  14.4% |  15.8% |  15.7%
Biased-BA(ResNet-18) [6] |  6.3% |  9.0% |  22.7% |  34.4% |  40.6%
SQBA(Inception-V3) |  57.6% |  62.8% |  68.4% |  72.4% |  75.8%
|  SQBA(ResNet-18) |  65.5% |  69.2% |  73.3% |  78.0% |  80.9%

Table 6: Query Budgets and ASR of hard-label based black-box attacks on VGG-16
trained on ImageNet

The experiments were conducted over two standard image datasets: CIFAR-10 [23]
and ImageNet-100 [11]. CIFAR-10 dataset is composed of RGB image examples in
the $3\times 32\times 32$ dimensional spaces. It has 10 classes with 6,000
examples per class. ImageNet-100 is a subset of ImageNet-1k dataset. It
consists of random 100 classes out of 1K classes of the full dataset, and each
class has 1,350 examples. All the models used in the experiments are listed in
Table 2. The models were off the shelf, which were not re-trained nor
modified, obtained from publicly available libraries, such as Zenodo [1] for
CIFAR-10 and Pytorch torchvision library [3] for ImageNet-1K models. All the
examples were transformed as instructed by the libraries. For example,
examples in ImageNet-100 dataset were rescaled to $3\times 256\times 256$
dimensional space, and centre cropped to $3\times 224\times 224$ dimensional
RGB examples. From the validation split of each dataset, dedicated subsets
were created with the examples correctly classified by the target models to
avoid artificial inflation of the success rate. 1000 examples were randomly
down sampled further from the subsets to be used in the evaluation. As the
objective dataset, ImageNet-100, is a direct subset of ImageNet-1K dataset,
DNN models trained on ImageNet-1K dataset are naturally ready for the
objective dataset. In the rest of the report, ImageNet-100 dataset is
presented as ImageNet unless otherwise specified.
For fair and easy comparison, the DNN models, which are publicly well known
and pre-trained with the datasets described above, were chosen for the
experiments. Specifically, for the experiments with CIFAR-10 dataset, two DNN
models, ResNet-18 and MobileNet-V2 were used as the target models. SQBA attack
method conducted two attacks on each target model utilising different
surrogate models, whose architectures are different from others, to
demonstrate the generalisation of the method. To attack ResNet-18 target
model, MobileNet-V2 and GoogLeNet were chosen for the surrogate model, while
Inception-V3 and ResNet-18 were used to attack MobileNet-V2. For the
experiments with ImageNet dataset, VGG-16 and EfficientNet-b3 were attacked by
the methods. SQBA attack method utilised ResNet-18 and EfficientNet-b3 models
to perform attacks on VGG-16 target model. ResNet-18 and VGG-16 models were
used as the surrogate models to attack EfficientNet-b3 target model. Each
group of models used in the experiments are detailed in Table 2.

###  5.3 Adversarial Attack Budgets

In order to generate meaningful adversarial examples, it is necessary to
impose reasonable constraints to the attack. An unconstrained attack perhaps
achieve a successful attack by, for example, disturbing target model with
tremendous number of queries, or introducing large and brutal perturbations to
an input example that would alter its semantics [7]. Since such attacks are
not considered to satisfy the goal of adversary, which is an adversarial
example shall have human-imperceptible perturbation, two restrictions were
introduced in the experiments. In the rest of this report, the restrictions
are called attack budgets.

The first attack budget used in order to quantify the performance of the
attack methods is a metric used in [12] as:

$\rho(\tilde{x}):=\frac{\|\mu\|_{2}}{\|x\|_{2}}$ (12)

where $\mu$ is the perturbation vector added to the original example $x$ to
achieve a successful adversarial attack. This metric was originally used to
measure the average robustness. In this report, however, $\rho(\tilde{x})$ is
used as the perturbation budget to determine the successfulness of an
individual attack. $\rho(\tilde{x})$ in fact indicates the distance between
$x$ and $\tilde{x}$, therefore the lower value means the higher robustness.

CIFAR-10: ResNet-18-AT at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  750 |  1000
Attack Methods |  HSJA [8] |  16.5% |  22.1%
Sign-Opt [10] |  8.6% |  9.5%
Biased-BA(GoogLeNet) [6] |  3.0% |  5.9%
SQBA(GoogLeNet) |  18.0% |  22.5%

CIFAR-10: MobileNet-V2-AT at $\rho(\tilde{x})\leq 0.10$
---
Query Budget |  750 |  1000
Attack Methods |  HSJA [8] |  24.5% |  27.6%
Sign-Opt [10] |  12.1% |  13.5%
Biased-BA(Inception-V3) [6] |  10.5% |  25.1%
SQBA(Inception-V3) |  25.2% |  31.9%

Table 7: Comparison of black-box attacks on defended models for CIFAR-10
dataset

As the second attack budget, the number of queries that an attack method can
use is also restricted by the five different budgets such as $\\{$100, 250,
500, 750, 1000$\\}$. The highest allowed number of queries 1000 may seem to be
in-efficient. However, some object classification applications which utilise
modern techniques such as GPU or FPGA accelerated systems are already capable
of processing more than 1000 images per second [2].

###  5.4 Attacks on CIFAR-10 Dataset

In this section the results of black-box adversarial attacks with CIFAR-10
dataset are reported. Pre-trained ResNet-18 and MobileNet-V2, which achieves
93.07% and 93.90% classification accuracies respectively, were chosen for the
target models. All the attacks were limited with the perturbation budget
$\rho(\tilde{x})<0.1$, and to at most the five query budgets as described in
Section 5.3. Any successful attack achieved with queries or perturbation which
exceeded the budgets was reported as a failed attack.

As shown in Table 2, Two SQBA with different surrogate models performed black-
box attacks to each target model, and their performance was compared with four
state-of-the-art hard-label based attack methods listed in Section 5.1. The
attack results over ResNet-18 target model is presented in Table 4. SQBA with
both MobileNet-V2 and GoogLeNet surrogate models outperformed benchmark
methods across all the query budgets, specifically SQBA with MobileNet-V2
showed the best attack performance achieving 82.0% ASR at 1000 query budget.
Especially in the very small query scenarios, where queries are limited with
100 and 250 at most, SQBA attacks performed approximately 4 and 5 times higher
ASR respectively compared with the best performing benchmark. Table 4 details
attack performances on MobileNet-V2 target model. SQBA attacks also
outperformed benchmarks in this experiment. At 100 query budget, both SQBA
attacks with Inception-V3 and ResNet-18 surrogate models achieved over 70.0%
ASR while other bench mark methods didn’t show meaningful attack performances
yet.

###  5.5 Attacks on ImageNet Dataset

This section details the performances of black-box adversarial attacks
conducted against target-models trained on ImageNet dataset. For the
experiments, pre-trained EfficientNet-b3 and VGG-16 DNN models were chosen for
the target models, and they respectively achieve 83.58% and 78.98%
classification accuracies over ImageNet-100 dataset. Surrogate models used in
SQBA attacks are detailed in Table 2. The benchmark methods and attack budgets
used in CIFAR-10 experiments were also applied in this series of experiments
to evaluate the performance of the SQBA methods.

Table 6 and Table 6 show the performances of the attack methods over ImageNet
models, EfficientNet-b3 and VGG-16 respectively. While it is a general
observation that SQBA attacks outperformed all other methods, in detail SQBA
attacks achieved approximately 5 times higher ASR compared with the benchmarks
at small query budgets 100 and 250, interestingly all methods struggled to
achieve successful attacks to EfficientNet-b3 model. At 1000 query budget,
HSJA [8] performed 40% degraded ASR compared to the same attack to VGG-16
target model, while SQBA with ResNet-18 achieved 32% degraded ASR compared
with the attack on VGG-16.

###  5.6 Attacks on Defended models

Attack methods were further evaluated with the defended models. ResNet-18-AT
and MobileNet-V2-AT models have identical architectures to the original models
used in Section 5.4. They were trained with CIFAR-10 dataset using the PGD
adversarial training method [26] to achieve 67.24% and 68.49% standard
classification accuracies respectively. Table 7 show the comparison of black-
box attacks on both defended models. In these experiments, two query budgets,
750 and 1000, were considered, and BA attack method which didn’t show
meaningful ASR in the previous experiments was not included. GoogLeNet and
Inception-V3 were chosen for the surrogate models to attack ResNet-18-AT and
MobileNet-V2-AT respectively. It can be observed that the defended models
successfully reduced the ASR of all the attack methods to approximately 70.0%
less than their attacks on the original models. Although SQBA attacks showed
the improved performances compared with the benchmarks, their performance was
also heavily degraded achieving 22.5% on ResNet-18-AT and 31.9% on
MobileNet-V2-AT target models at 1000 query budget, which were 71.5% and 66.6%
respectively less ASR compared with the same attacks on the original models.

##  6 Conclusion

SQBA method which combines transfer based and hard-label based adversarial
attacks was introduced. SQBA is designed to generate likely adversarial
examples for an unknown classifier whose predicted class is only accessible.
The method takes an advantage of the transfer based algorithm for the rapid
search of an initial convergence point which is followed by the guided
optimisation procedure. Through the experiments it was demonstrated that SQBA
is capable of achieving higher ASR with smaller query budgets compared with
the benchmarks. One drawback of black-box attack methods which utilise
transfer based algorithms degrades the performance when gradients from the
transfer based algorithms disagree with the target model, and SQBA is not
immune from this problem. Future work may focus on improving the adversarial
transferability in the SQBA method to address this issue.

## References

  * [1] https://github.com/huyvnphan.
  * [2] https://github.com/u39kun/deep-learning-benchmark.  Technical report.
  * [3] https://pytorch.org/vision/stable/models.html.
  * [4] A. N. Bhagoji, W. He, B. Li, and D. Song.  Practical black-box attacks on deep neural networks using efficient query mechanisms.  The European Conference on Computer Vision, 2018.
  * [5] W. Brendel, J. Rauber, and M. Bethge.  Decision based adversarial attacks: Reliable attacks against black-box machine learning models.  International Conference on Learning Representation, 2018.
  * [6] T. Brunner, F. Diehl, M. T. Le, and A. Knoll.  Guessing smart: Biased sampling for efficient black-box adversarial attacks.  arXiv preprent arXiv:1812.09803v3, 2019.
  * [7] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin.  On evaluating adversarial robustness.  arXiv preprint arXiv:1902.06705, 2019.
  * [8] J. Chen, M. I. Jordan, and M. J. Wainwright.  Hopskipjumpattack: A query-efficient decision-based attack.  IEEE Symposium on Security and Privacy, 2020.
  * [9] P. Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C. J. Hsieh.  Zoo: Zeroth order optimisation based black-box attacks to deep neural networks without training substitute models.  ACM Workshop on Artificial Inteligence and Security, 2017.
  * [10] M. Cheng, S. Singh, P. Chen, P.Y. Chen, S. Liu, and C.J. Hsleh.  Sign-opt: A query-efficient hard-label adversarial attack.  International Conference on Learning Representations, 2020.
  * [11] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  IEEE Conference on Computer Vision and Pattern Recognition, 2009\.
  * [12] S.M.M. Dezfooli, A. Fawzi, P. Frossard, and E.P.F. Lausanne.  ”deepfool”: A simple and accurate method to fool deep neural networks.  IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [13] Y. Dong, S. Cheng, T. Pang, H. Su, and J. Zhu.  Query-efficient black-box adversarial attacks guided by a transfer-based prior.  arXiv:2203.06560v1, 2022.
  * [14] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li.  Boosting adversarial attacks with momentum.  IEEE Conference on Computer Vision and Pattern Recognition, 2018\.
  * [15] Y. Dong, T. Pang, H. Su, and J. Zhu.  Evading defenses to transferable adversarial examples by translation-invariant attacks.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019\.
  * [16] A. Galstyan and P. R. Cohen.  Empirical comparison of ”hard” and ”soft” label propagation for relational classification.  The 17th International Conference on Inductive Logic Programming, 2007.
  * [17] I. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  International Conference on Learning Representations, 2015.
  * [18] C. Guo, J. R. Gardner, Y. You, A. G. Wilson, and K. Q. Weinberger.  Simple black-box adversarial attacks.  International Conference on Learning Representation, 2019.
  * [19] Z. Huang and T. Zhang.  Black-box adversarial attack with transferable model-based embedding.  International Conference on Learning Representations, 2020.
  * [20] A Ilyas, L Engstrom, A Athalye, and J Lin.  Black-box adversarial attacks with limited queries and information.  International Conference on Machine Learning, 2018.
  * [21] A. Ilyas, L. Engstrom, and A. Madry.  Prior convictions: Black-box adversarial attacks with bandits and priors.  International Conference on Learning Representations, 2019.
  * [22] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimizastion.  International Conference on Learning Representations, 2015.
  * [23] A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  Techinical Report, Citeseer, 2009.
  * [24] H. Li, X. Xu, X. Zhang, S. Yang, and B. Li.  Qeba: Query-efficient boundary-based blackbox attack.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020\.
  * [25] J. Lin, C. Song, K. He, L. Wang, and J. E. Hopcroft.  Nesterov accelerated gradient and scale invariance for adversarial attacks.  International Conference on Learning Representations, 2020.
  * [26] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu.  Towards deep learning models resistant to adversarial attacks.  International Conference on Learning Representations, 2018.
  * [27] N. Papernot, P. McDaniel, and I. Goodfellow.  Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.  arXiv preprint arXiv:1605.07277, 2016.
  * [28] N Papernot, P McDaniel, I Goodfellow, S Jha, Z.B Celik, and A Swami.  Practical black-box attacks against machine learning.  Proceedings of the 2017 ACM on Asia conference on computer and communicaitons security, 2017.
  * [29] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.  Intriguing properties of neural networks.  International Conference on Learning Representations, 2013.
  * [30] F. Tramer, N. Papernot, I. Goodfellow, D. Boneh, and P. D. McDaniel.  The space of transferable adversarial examples.  arXiv preprint arXiv:1704.03453, 2017.
  * [31] C. Tu, P. Ting, P. Chen, S. Liu, H. Zhang, J. Yi, C. Hsieh, and S. Cheng.  Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks.  The Thirty-Third AAAI Conference on Artificial Intelligence, 2019\.
  * [32] Xiaosen Wang and Kun He.  Enhancing the transferability of adversarial attacks through variance tuning.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021\.
  * [33] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. Yuille.  Improving transferability of adversarial examples with input diversity.  IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [34] J. Yang, Y. Jiang, X. Huang, B. Ni, and C. Zhao.  Learning black-box attackers with transferable priors and query feedback.  Conference on Neural Information Processing System, 2020.
  * [35] L. Zhou, P. Cui, X. Zhang, Y. Jiang, and S. Yang.  Adversarial eigen attack on black-box models.  arXiv:2009.00097, 2020.

##  7 Supplemental Material

###  7.1 Dual Gradient Attack Method (DGM)

This supplemental material describes Dual Gradient white-box attack
Method(DGM) which is adopted in SQBA attack method. DGM uses dual gradient
vectors to effectively find perturbations. The procedure of DGM attack
consists of two stages: (1) Generating adversarial perturbation, and (2) Fine-
tuning generated perturbation.

####  7.1.1 Adversarial Perturbation

An input example $x\in\mathbb{R}^{m}$ to be estimated by the $k$-class
classifier $F(x)$ can be seen as a data point located in a convex polyhedron
[12], whose faces represent classes $c_{i}$, where $i=0,1,...,k$. The
orthogonal distance from a class $c_{i}$ to the data point $x$ denotes the
quantified classification probability, and it is expressed as
$\Delta(x;c_{i})$. The decision boundary of the classifier can be seen as an
affine linear equation also in the polyhedron. Since the primary objective of
adversarial attack is to modify input example $x$ to mislead a target model,
the true class $c^{\dagger}$ and an adversarial class $\tilde{c}$ need to be
considered.

The orthogonal distance between $c^{\dagger}$ and $x$ is smaller than other
classes in the polyhedron as
$\Delta(x;c^{\dagger})<\Delta(x;c^{i}),\;\forall\;c^{\dagger}\neq c^{i}$. To
mislead the target model, therefore, $\Delta(x;c^{\dagger})$ needs to be
increased by moving example $x$ sufficiently to an adversarial region where
belongs to another class $c^{i}$. DGM searchs the optimal path to transport
$x$ by utilising two vectors, $g^{-}$ and $g^{+}$, which are calculated with
respect to the true class $c^{\dagger}$ and a potential adversarial class
$c^{i}$ respectively. The vector $g^{-}$ has the negative direction to the
class, therefore, it moves $x$ away from the true class. In contrast the
positive directional vector $g^{+}$ transports $x$ closer to the adversarial
class. The gradient vectors are calculated as:

$g^{+/-}=\left.\begin{cases}l_{2}\;\;:\nabla_{F}f_{c}(x)/\mbox{max}\left(\nabla_{F}f_{c}(x)\right)\\\
l_{\infty}\,:\mbox{sign}\left(\nabla_{F}f_{c}(x)\right)\end{cases}\right.$
(13)

where $\nabla_{F}f_{c}(x)$ is a gradient vector obtained from the backward
process of $F(x)$ with input example $x$ and associated class $c$. DGM
iteratively conducts such process to efficiently find the optimal direction of
adversarial perturbation vector $\mu_{t}$ as:

$\mu_{t}=(-\alpha_{t})g^{-}_{t}+(1-\alpha_{t})g^{+}_{t}$ (14)

where $\alpha_{t}\in[0,0.3]$ is a penalty parameter applied to both
directional vectors. The penalty parameter is used to help the stable
convergence in searching an optimal vector $\mu_{t}$, and updated in every
iteration as:

$\alpha_{t}=\min\left(1/e^{4\lambda},\;0.3\right),\quad\text{where}\;\lambda=\frac{f_{c^{i}_{t}}(x^{\prime}_{t})}{\left(f_{c^{\dagger}}(x^{\prime}_{t})+f_{c^{i}_{t}}(x^{\prime}_{t})\right)}$
(15)

Finally an intermediate adversarial example is calculated with a scaling
factor $\epsilon\leq 1$, which is a small positive value as

$x^{\prime}_{t+1}=\text{clamp}\left(x^{\prime}_{t}+\epsilon\mu_{t}\right)\big{|}_{[\text{min}(x),\text{max}(x)]}.$
(16)

Algorithm 2 outlines the process to find an adversarial example with DGM
method.

$\mbox{{input:} example $x$, true class $c^{\dagger}$, classifier }f$

output: adversarial example $\tilde{x}$

while TRUE do

$\\{c^{0}_{t},c^{1}_{t},..\\}=\;$ Sort$(f(x^{\prime}_{t}),\;descend)$

if $c^{0}_{t}\neq c^{\dagger}$ then Break; end

$\tilde{c}=c^{1}_{t}$

$\alpha_{t}=\;$ Equation (15) $\leftarrow\tilde{c},c^{\dagger}$

$g^{+}_{t}=\;$ Equation (13) $\leftarrow\tilde{c}$

$g^{-}_{t}=\;$ Equation (13) $\leftarrow c^{\dagger}$

$\mu_{t}=\;$ Equation (14)

$x^{\prime}_{t+1}=\;$ Equation (16)

$t=t+1$

end while

$\tilde{x}=\;$ Tune$(x^{\prime}_{t})$

Algorithm 2 DGM Adversarial Example Computation

input: example $x$, initial adversary $x^{\prime}$,

true class $c^{\dagger}$, classifier $f$

output: adversarial example $\tilde{x}$

$t=0,x^{\prime}_{t}=x^{\prime}$

while TRUE do

$\tilde{x}=x^{\prime}_{t}$

$\mathcal{J}=\;$ MSE$(x,x^{\prime}_{t})$

$x^{\prime}_{t}=\;$ ADAM$(x^{\prime}_{t},\mathcal{J})$

$c^{\prime}_{t}=\operatorname*{arg\,max}[f(x^{\prime}_{t})]$

if $c^{\prime}_{t}==c^{\dagger}$ then Break; end

$t=t+1$

end while

$\tilde{x}=x^{\prime}_{t}$

Algorithm 3 DGM Adversarial Example Tuning Figure 3: Examples of targeted
attack. DGM $l_{2}$ attack is applied to the CIFAR-10 dataset performing the
targeted attack for each source/target pair. First column is the clean images

####  7.1.2 Adversary Tuning

Attack method discussed in the previous section focuses on the effectiveness
in finding an adversarial example $\tilde{x}$, which successfully leads to the
misclassification with a high classification probability yet. One other
objective of the attack is finding the minimum perturbation to make the
adversarial example $\tilde{x}$ sufficiently close to the input example $x$.
Input example can be seen as a fixed data point. Therefore, the goal of this
optimisation is to find $\delta$ that minimises $l_{2}$ distance between $x$
and $\tilde{x}$. DGM solves such problem by formulating a simple objective of
the iterative optimisation with Mean Square Error (MSE) as:

$\text{min}\left(\text{MSE}(x,\tilde{x})\right),\quad\text{such
that}\;\tilde{x}\in[\text{min}(x),\text{max}(x)]^{m}$ (17)

where $\tilde{x}=x+\delta$ is an adversarial example. To achieve the goal to
find $\delta$ that minimises $\text{MSE}(x,\tilde{x})$, ADAM [22] optimiser is
deployed in DGM method as detailed in Algorithm 3.

####  7.1.3 Source Code

The source code to reproduce experimental results can be found from
https://github.com/jpark04-qub/SQBA