# A Generative Adversarial Attack for Multilingual Text Classifiers

Tom Roth1,2, Inigo Jauregi Unanue1,3, Alsharif Abuadbba2, Massimo Piccardi1

###### Abstract

Current adversarial attack algorithms, where an adversary changes a text to
fool a victim model, have been repeatedly shown to be effective against text
classifiers. These attacks, however, generally assume that the victim model is
monolingual and cannot be used to target multilingual victim models, a
significant limitation given the increased use of these models. For this
reason, in this work we propose an approach to fine-tune a multilingual
paraphrase model with an adversarial objective so that it becomes able to
generate effective adversarial examples against multilingual classifiers. The
training objective incorporates a set of pre-trained models to ensure text
quality and language consistency of the generated text. In addition, all the
models are suitably connected to the generator by vocabulary-mapping matrices,
allowing for full end-to-end differentiability of the overall training
pipeline. The experimental validation over two multilingual datasets and five
languages has shown the effectiveness of the proposed approach compared to
existing baselines, particularly in terms of query efficiency. We also provide
a detailed analysis of the generated attacks and discuss limitations and
opportunities for future research.

##  1 Introduction

Advancements in machine learning have led to models that achieve remarkable
performance across a variety of natural language processing (NLP) tasks.
Despite this progress, these models remain susceptible to adversarial attacks,
which manipulate inputs to induce incorrect predictions while maintaining
linguistic coherence and fluency. In this work we consider text
classification, for which a wide variety of adversarial attacks have been
developed, ranging from simple character or token replacements to fully
generative models.

Text classification models can be developed to process only a specific
language (i.e., they are monolingual) or to process multiple languages (i.e.,
they are multilingual). As processing multiple languages with a single model
is more scalable and efficient than training a number of monolingual models,
multilingual models are now widely used across many NLP applications. Despite
this, multilingual models have been largely overlooked as victims of
adversarial attacks, with the majority of existing attack algorithms being
implicitly monolingual and ill-suited to handle multiple languages.

The most common approach for creating text adversarial examples is to leverage
combinatorial optimisation in some form. In this approach, an attack takes in
input a correctly-classified example and attempts to transform it into an
adversarial one by iteratively modifying its individual tokens. The
modifications adhere to a set of allowed transformations, such as word
insertion or replacement, and are guided by a chosen objective function and
search method. Constraints are also imposed to ensure the validity of the
transformations, such as requiring semantic similarity to the original input.
The process typically stops once a successful adversarial example is found, or
the maximum number of attempts is reached. De facto, these attacks solve a
separate optimisation problem for each example and are highly effective —
provided that they are allowed enough steps, time and computational resources.
However, they are typically slow and hard to scale, and the space of the
adversarial examples is constrained by the choice of the allowed
transformations, which may limit their richness and diversity.

An alternative approach is to use a trainable generative model to produce the
adversarial examples. These models learn a mapping from an original example as
input to an adversarial example as output based on a training dataset. At
inference time, they are able to produce an adversarial attack from the given
input in a single forward pass. Since these models do not perform an
optimisation process for each input example, they are, in principle, unlikely
to match the performance of the transformation-based approach. However, the
generative approach mitigates many of its disadvantages: 1) producing the
adversarial example is much faster, requiring only a single forward pass per
batch of examples; 2) multiple adversarial candidates can be generated
simultaneously just by using beam decoding; and 3) it can potentially perform
richer and more diverse transformations of the input example.

With this motivation, in this paper we propose an approach for training a
generative model to attack a multilingual victim model, under a white-box
assumption111Which holds for all publicly-distributed models, such as those on
Hugging Face and GitHub.. We begin with a generative pre-trained multilingual
model (specifically, mT5 (Xue et al. 2021)) capable of generating text across
more than a hundred languages. Although pre-trained, this model does not yet
solve any task, and so we first train it with a multilingual paraphrasing
objective. This allows the model to first learn a diverse range of text-to-
text transforms in multiple languages. As second stage of training, we fine-
tune the generative model with an adversarial training objective. This
training objective incorporates both the victim model itself, and also a
number of other pre-trained models that help maintain the linguistic
properties of the text. Amongst them, a language detection model prevents the
generative model from outputting text in languages different from the
original’s, which is an implicit requirement of this task and a key
contribution of the proposed approach. The models are connected with the
generative model by matrices that remap their vocabularies, allowing the use
of any models in the training objective, while at the same time retaining the
end-to-end differentiability of the pipeline. At inference time, the model
receives an input in language $L$ and generates adversarial candidates in the
same language, provided $L$ is amongst the languages it was fine-tuned in.

To comparatively evaluate the performance of the proposed approach, we have
adapted two existing monolingual optimisation attack algorithms to the
multilingual case, and used them as baselines. As datasets, we have used two
multilingual datasets that incorporate four and five languages, respectively.
The experimental results show that our generative model is a fast, diverse and
effective attacker, and matches the performance of baselines that run for far
longer. In summary, our paper makes the following main contributions:

  1. 1.

proposes a generative attack for multilingual victim models (a first to the
best of our knowledge);

  2. 2.

proposes a dedicated training objective that leverages the victim model itself
and various other modules that promote desirable linguistic properties;

  3. 3.

makes use of vocabulary-mapping matrices to allow maximum flexibility in the
choice of the models and ensure differentiability end-to-end;

  4. 4.

lastly, our experimental results over two multilingual datasets and five
languages show a remarkable comparative performance against the chosen
baselines.

##  2 Related Work

Combinatorial optimisation attacks. Combinatorial optimisation attacks are
undoubtedly the most common type of attack for text classifiers. Many attacks
have been proposed, each defined by their own allowed transformations, search
method, and constraints. Well-known attacks include TextFooler (Jin et al.
2020), BERTAttack (Li et al. 2020), BAE (Garg and Ramakrishnan 2020) and CLARE
(Li et al. 2021). The most recent research has focused on a range of aspects,
such as increasing the readability of the adversarial examples (Chang et al.
2023), improving the query efficiency of the attacks (Zhu, Zhao, and Wu 2023;
Lee et al. 2022) and adapting adversarial attacks designed for images to the
text domain (Yuan et al. 2023).

Generative attacks. Although less explored, a number of generative approaches
have been proposed. The most relevant to our work are the variations on the
white-box, differentiable model-cascading design described in the
Introduction. Such work includes Wang et al. (2020), which has incorporated a
downstream model to control the topic of the generated adversarial candidates
at inference time, and Xu et al. (2021), which has incorporated several
features (e.g., label smoothing and copy mechanisms) to improve the quality of
the generated examples. In turn, Guo et al. (2021) have used pre-trained
models to learn an example-specific matrix of token probabilities, which can
be repeatedly sampled to generate adversarial examples. To avoid the use of
vocabulary-mapping matrices (or similar) previous work has largely assumed a
scenario where all the component models have to share the same vocabulary,
which is a very restrictive assumption. An exception is Song et al. (2021),
which has constrained the generative model to only output common tokens to all
vocabularies. However, this is also very restrictive.

Multilingual attacks. Most existing text adversarial attacks are monolingual,
and predominantly in English. Some work has partially addressed the second
restriction by designing attacks for other languages, such as Chinese (Liu,
Cai, and Qi 2023; Wang et al. 2022) or Arabic (Alshemali and Kalita 2019). The
only multilingual attack we are aware of is Rosenthal, Bornea, and Sil (2021),
who have attacked a multilingual question answering system by adding
distracting statements to the question. Other multilingual attacks have been
developed for the related task of code-mixing, where a single input contains
multiple languages (mimicking how multilingual speakers sometimes mix multiple
languages when speaking). Code-mixing attacks apply various perturbations to
the example: Tan and Joty (2021) have introduced words and phrases through the
use of bilingual dictionaries and translation models, and Das et al. (2021)
have made phonetically-similar replacements for the Bengali-English and Hindi-
English scenarios. However, the code-mixing scenario is different from what we
target in our paper: in our case, each example in input to the multilingual
classifier is assumed to be in only one language. This allows enforcing a
language consistency constraint as a key requirement for the acceptability of
the attack.

##  3 Proposed Approach

The training approach fine-tunes a pre-trained generative model $g$ so that it
can generate adversarial examples for a victim model $v$. Besides the victim,
the approach incorporates two additional component models into the training
objective: a semantic similarity model $s$ and a language detector model $l$.
All models involved have an associated tokeniser with a specific vocabulary,
which for model $i$ is denoted by $V_{i}$. The parameters of all models are
fixed during training, except for $g$. After fine-tuning, $g$ is able to
generate adversarial examples for $v$. The training setup is shown in Figure
1.

Figure 1: The proposed approach. The generative model has been trained using a
loss function comprised of a number of factors, including scores from three
component models (victim, similarity, and language detection), a KL divergence
score, and a diversity score.

###  3.1 Pre-Training

For our generative model, we start from a pre-trained mT5-base (Xue et al.
2021) checkpoint. This model has been trained using a span-corruption
objective, and has not been fine-tuned on any downstream task, so the first
pre-training step we take is to fine-tune it on a multilingual text
paraphrasing training dataset. After this training step, the model is able to
generate a range of diverse paraphrases across multiple languages. This gives
the starting weights for the generative model $g$, that is later fine-tuned.
We also take a copy of this model and fix its weights to serve as the
reference model $g^{*}$ (see Section 3.4).

###  3.2 Fine-Tuning

During the forward pass, the input is an original example $x$ which is passed
to $g$, generating output $x^{\prime}$ with length $T$. The corresponding
sequence of token probability distributions forms a matrix $P$, with
dimensions $T\times|V_{g}|$.

The generated text cannot be passed to the objective models as is, as that
would involve non-differentiable operators (either sampling or argmax).
Therefore, we employ a different approach to retain the differentiability of
the pipeline: passing a weighted average of token embeddings to the downstream
models. This is created by combining $P$ with the token embedding matrix of
the downstream model, and a second matrix that maps the vocabulary of the
generative model to the vocabulary of the downstream model. We call this
second matrix a vocabulary-mapping matrix and describe it in detail in Section
3.3.

More formally, for any component model $V_{i}$, with $i\in\\{v,s,l\\}$, we
have computed the respective weighted embeddings $W_{i}$ as:

$W_{i}=PM_{i}E_{i}$

where $E_{i}$ is the token embedding matrix of model $i$, and $M_{i}$ is the
vocabulary-mapping matrix that maps $V_{g}$, the vocabulary of $P$, to
$V_{i}$, the vocabulary of model $i$.

As done in prior research (Xu et al. 2021; Wang et al. 2020), we have used the
Gumbel-softmax reparametrisation trick (Jang, Gu, and Poole 2017) and replaced
$P$ with a sampled matrix $P_{b}$ that has incorporated $\text{Gumbel}(\tau)$
noise. Here, $\tau$ is a temperature parameter that controls the entropy of
the resulting probability distributions. This step introduces stochasticity
into the token distribution with the aim to increase exploration during
training.

Afterwards, the weighted embeddings $W_{i},i\in\\{v,s,l\\}$ are passed to
their respective downstream models, and the output from each downstream model
is used in the loss function (Section 3.4). The backward pass computes the
chained gradients and updates the parameters of the generative model, using
standard backpropagation.

###  3.3 Vocabulary-Mapping Matrices

Although required for our differentiable training approach, it is not
straightforward to map tokens between vocabularies, particularly when they are
of different sizes and have been constructed with different tokenisation
algorithms. In our implementation, the vocabulary of the generative paraphrase
model is constructed using the SentencePiece (Kudo and Richardson 2018)
tokenisation algorithm, while the downstream models’ vocabularies use
WordPiece (Wu et al. 2016), and differ in content and size. We have been able
to construct a workable mapping with the following rules:

  1. 1.

Map one-to-one all direct matches between SentencePiece start-of-word tokens
and WordPiece non-continuation tokens.

  2. 2.

Map one-to-one all direct matches between SentencePiece non start-of-word
tokens and WordPiece continuation tokens.

  3. 3.

Map any special tokens (e.g., PAD, EOS, UNK) directly across both
vocabularies. Map the extra-id tokens in the generator’s vocabulary to the UNK
WordPiece token.

  4. 4.

Map the remaining SentencePiece tokens one-to-many with WordPiece tokens using
the WordPiece tokeniser, stripping any generated special tokens, and assigning
equal probabilities to all matches. Map any remaining tokens (e.g. special
cases like \xad) to the UNK token.

The end result is a matrix $M_{i}$ that maps tokens from the vocabulary of the
generative model, $V_{g}$, to $V_{i}$, the vocabulary of the downstream model.
In $M_{i}$, each row is a probability distribution that represents the one-to-
many token mapping, with the sum of each row being 1, and the vast majority of
entries being 0. The shape of $M_{i}$ is $|V_{g}|\times|V_{i}|$, and as this
is a large matrix222$|V_{g}|$ alone is 250K., to save space we store it as a
sparse matrix.

###  3.4 Loss Function

The training objective, denoted as $o(x,x^{\prime})$, has been designed to
balance the adversarial and linguistic factors required by a successful
multilingual adversarial attack. It is given by

$\begin{split}o(x,x^{\prime})=\alpha_{v}t(v(x,x^{\prime}),\beta_{v})+\\\
\alpha_{s}t(s(x,x^{\prime}),\beta_{s})+\\\
\alpha_{l}t^{*}(l(x,x^{\prime}),\beta_{l})-\\\
\alpha_{KL}t^{*}(D_{KL}(x,x^{\prime}),\beta_{KL})\end{split}$ (1)

where:

  * •

$v(x,x^{\prime})$ is the ‘victim model score’: the degradation in victim model
confidence of the true class when replacing $x$ with $x^{\prime}$.

  * •

$s(x,x^{\prime})$ is the ‘similarity score’ between $x$ and $x^{\prime}$,
given by the cosine similarity of their sentence embeddings from a pre-trained
multilingual Sentence-BERT model (Reimers and Gurevych 2019).

  * •

$l(x,x^{\prime})$ is the ‘language consistency score’, given by the
degradation in original-language confidence between $x$ and $x^{\prime}$,
using a pre-trained language detection model.

  * •

$D_{KL}$ is a Kullback-Leibler (KL) divergence term, calculated between the
token probabilities produced by the fine-tuned generative model $g$ and a
reference model $g^{*}$. This term prevents the fine-tuned distribution from
deviating too much from the reference, which we take as the multilingual
paraphrase model before fine-tuning, allowing the beneficial paraphrasing
qualities of $g^{*}$ to be maintained.333Due to memory constraints, $g^{*}$ is
only a part of the initial paraphrase model. $D_{KL}$ is defined as:

$\displaystyle D_{KL}=\frac{1}{T}\mathbb{E}_{x\sim\mathcal{D},x^{\prime}\sim
g(x;\theta)}[\log p_{g}(x^{\prime}|x)-$ (2) $\displaystyle\log
p_{g^{*}}(x^{\prime}|x)]$

In (2), the divergence has been normalised by the generated sequence length,
$T$, to prevent longer sequences from being unfairly penalised.

  * •

$t$ and $t^{*}$ are threshold clipping operators, with $t(a,\beta)=a$ if
$a<\beta$, and $0$ otherwise, and $t^{*}(a,\beta)=a$ if $a>\beta$, and 0
otherwise. These operators are intended to encourage a balanced optimisation
of the multiple objective components, with hyperparameters $\alpha$ and
$\beta$ controlling the contribution of each component.

This training objective $o(x,x^{\prime})$ is then incorporated into a batch-
level loss, $L$, which is defined as:

$\begin{split}L=-\left(\frac{1}{|B|}\sum_{(x,x^{\prime})\in
B}o(x,x^{\prime})\right)+\alpha_{d}\hskip 2.0ptd(B)\end{split}$ (3)

where $d(B)$ is a batch-level diversity score, and $\alpha_{d}$ its
corresponding coefficient.

To compute $d(B)$, within batch $B$ and using the same model as for the
similarity score, we calculate two cosine similarity matrices: one of the mean
of the token embeddings for each generated example, and one of the embeddings
of the original input text. We then take the mean of the squared difference of
the upper triangular matrices as the diversity penalty $d(B)$. This term
encourages the generated examples to maintain a similar batch diversity as for
the original examples, as to prevent degeneration during training and improve
the variety of the generated examples.

As each individual component in the loss function is differentiable and their
combination is linear, the overall loss function is also differentiable and
the system can be efficiently optimised using backpropagation. The overall
design is flexible, and each coefficient can be adjusted to prioritise
different objectives, such as attack strength, textual fluency or language
consistency.

###  3.5 Validation and Early Stopping

The quality of the generated text and its attack strength are somehow
inversely proportional to the duration of training: shorter training yields
high-quality text with lower attack strength, while prolonged training
increases attack strength, yet degrades text quality. Deciding when to stop
training is hence an important decision, and we have used the following
stopping criteria: during the validation phase, we generate 16 candidates per
original example, decoding using diverse beam search (Vijayakumar et al.
2016). Each candidate is then given a ‘score’ based on its adversarial and
linguistic qualities:

$\gamma_{v}v(x,x^{\prime})+Q(x,x^{\prime}),$

with

$Q(x,x^{\prime})=\gamma_{s}s(x,x^{\prime})-\gamma_{l}l(x,x^{\prime})-\gamma_{KL}D_{KL}(x,x^{\prime})$

being a ‘text quality’ score. The $\gamma$ coefficients set the balance
between attack strength and text quality.

The validation metric we use is the average score across all candidates for
the examples in the validation set. We calculate the metric after every
multiple of training steps, and halt the training process once the metric
fails to improve for a number of steps, as standard in early stopping.

##  4 Experimental Setup

###  4.1 Datasets and Models

We have evaluated the approach on two multilingual datasets. The first is the
Multilingual Amazon Reviews Corpus (MARC) (Keung et al. 2020), comprised of
Amazon customer ratings of the goods they purchased. For our experiments, we
have used the English, German, French, and Spanish language splits. The second
dataset is the Tweet Sentiment Multilingual (TSM) dataset (Barbieri, Espinosa
Anke, and Camacho-Collados 2022). This is a multilingual sentiment analysis
dataset that categorises a variety of tweets as negative, neutral or positive.
For this dataset, we have used the above four language splits, plus Arabic.
Finally, to initially train the multilingual paraphrasing objective, we have
used the TaPaCo dataset (Scherrer 2020), selecting the splits that correspond
to all languages covered above. For all datasets, we have filtered out all
examples longer than 32 tokens, as these longer examples are difficult to
paraphrase effectively.

As multilingual victim models, we have fine-tuned a pre-trained multilingual
DistilBERT model (Sanh et al. 2019) on the MARC and TSM datasets separately,
using only the languages mentioned above. The performance of the resulting
trained models is close to the state-of-the-art performance for both datasets,
making the adversarial attacks relevant. Full details of datasets and models,
including the specific choice of component models, are in Appendix A.

###  4.2 Baselines

As no general multilingual adversarial attack currently exists, for the
baselines we have extended CLARE (Li et al. 2021) and BAE-R (Garg and
Ramakrishnan 2020), two existing monolingual combinatorial optimisation
attacks, to the multilingual case. We term these new multilingual variants
mCLARE and mBAE. Both attacks originally used a BERT masked language model
(Devlin et al. 2019) to generate the transform token candidates, and the
Universal Sentence Encoder (Cer et al. 2018) to assess sentence similarity. In
their multilingual extension, we have replaced them with a multilingual BERT
masked language model and the multilingual Universal Sentence Encoder (Yang et
al. 2020), respectively. We have also removed language-dependent constraints
from the attacks that originally forbade changing stop-words and restricted
attack targets to words with a certain part-of-speech tag. The final mCLARE
attack is able to replace, insert, or merge individual tokens, while mBAE can
only replace.

Since the attack performance for these baselines directly depends on the
number of times that the attack is allowed to query the victim model, to get
an accurate view of performance we have run the baselines a number of times,
each with a differing amount of maximum queries. The number of queries can
also be seen as a proxy for the runtime of the method, for which there is a
clear correlation (Yoo et al. 2020). However, the actual runtime directly
depends on the time per query, which is approximately 10x larger for mCLARE
than mBAE.

###  4.3 Candidate Selection

At inference time, our generative fine-tuned model is capable of generating,
in principle, multiple adversarial candidates per input example by using beam
search. However, for fairness we have opted to select only one candidate from
our model, because it may be otherwise unfair to compare with the
combinatorial optimisation attacks which return only one adversarial example
per input. To select our candidate, we start by decoding using diverse beam
search (Vijayakumar et al. 2018) to create $n$ outputs for each original
example. Then, out of those that flip the label, we select that with the
highest text quality score, $Q(x,x^{\prime})$, as the candidate (see Section
3.5 for details).

(a) Results for the MARC dataset.

(b) Results for the TSM dataset.

Figure 2: Results for the MARC dataset (above) and the TSM dataset (below), overall and split by language. The x-axis is the maximum number of victim model queries, and the y-axis is the validated success rate (VSR) of the attacks. For all methods, we report VSR values for two different fluency threshold (see Section 4.4). The black dots plot the values for the proposed approach (averaged across three random seeds), while the lines plot the values for the baseline methods, with blue for mBAE and red for mCLARE. The results show that the proposed approach has achieved remarkable VSR values in many cases, and has achieved an impressive trade-off between performance and number of queries. At least one of the baselines eventually surpasses its performance in many cases, but only after many more queries (512, 1024 or more).  Dataset |  |  |  |  | Label |
---|---|---|---|---|---|---
MARC | Orig | I love that it keeps the wine cold. | Positive |
mCLARE | I not love that it keeps the wine cold. | Negative |
|  Flu: -7.11 Lang: 0.84 Sim: 0.96 |  |  |  |
mBAE | I doubt that it keeps the wine cold. | Negative |
|  Flu: -7.60 Lang: 0.85 Sim: 0.92 |  |  |  |
Ours | In my opinion, it keeps the wine cold! | Negative |
|  Flu: -7.63 Lang: 0.78 Sim: 0.82 |  |  |  |
MARC | Orig | Une batterie neuve qui fait à peine mieux que mon ancienne batterie. Décevant. | Negative |
mCLARE | Une batterie neuve qui fait à peine mieux que mon ancienne batterie. Ravel. | Positive |
|  Flu: -5.24 Lang: 0.97 Sim: 0.97 |  |  |  |
mBAE | Une batterie neuve qui fait vraiment bon mieux que mon ancienne batterie. Décevant. | Positive |
|  Flu: -4.89 Lang: 0.96 Sim: 0.99 |  |  |  |
Ours | C’est une batterie neuve qui est bien mieux que mon ancienne batterie. | Positive |
|  Flu: -6.74 Lang: 0.99 Sim: 0.75 |  |  |  |
TSM | Orig | Heute Abend dann noch Haare färben und Sport machen :D | Positive |
mCLARE | Heute Abend dann noch Haare färben und Sport machen :Roman | Neutral |
|  Flu: -8.19 Lang: 0.88 Sim: 0.99 |  |  |  |
mBAE | Heute Abend dann noch Haare färben und Sport machen :Roman | Neutral |
|  Flu: -8.19 Lang: 0.88 Sim: 0.99 |  |  |  |
Ours | Was hältst du davon, heute Abend noch Haare färben und Sport | Neutral |
|  Flu: -6.58 Lang: 0.97 Sim: 0.71 |  |  |  |
Table 1: Examples of successful adversarial attacks generated by the various methods for three languages. The token changes of the baselines are highlighted in blue, and the values of the fluency, language consistency, and semantic similarity are reported below. The examples show that our method has been able to create richer, coherent adversarial examples than the baselines, which are limited to modifying individual tokens. To generate successful examples, the baselines often employ “tricks” such as inserting negatives (first example), forming ungrammatical sentences (e.g. bon mieux for mBAE in the second example, or the misformed emoji :Roman in the third) and changing the base language (Ravel in the second example).  Mode |  Text | Label
---|---|---
Orig |  These raw cashews are delicious and fresh. | Pos
$+$Sim |  I’ll like this raw cashew. | Neg
$-$Sim |  I’m going to eat these raw cashiers. | Neg
Orig |  I love that it keeps the wine cold. | Pos
$+$Sim |  In my opinion, it keeps the wine cold! | Neg
$-$Sim |  The wine is delicious. I’m going to like | Neg
Table 2: Illustrative English examples from training with and without the semantic similarity component. In the first example, cashew is changed to cashier — a simple change of one subword token that loses the sentence meaning. In the second example, the information about the temperature of the wine is lost. Mode |  Text | Label
---|---|---
Orig |  Super, hilft schnell | Pos
$+$LD |  Irgendetwas, das ist schnell. | Neg
$-$LD |  Solch ein ayuda! | Neg
Orig |  Ces sachets sentent le plastique. | Neg
$+$LD |  Ces sachets sont pour le plastique! | Pos
$-$LD |  Ces sachetsgefühlt le plastique. | Pos
Table 3: Training without the language detection component often leads to
foreign words in the generated text. In the first example, the model has
directly translated the German hilft (to help) to the Spanish word ayuda.
Similarly, in the second example the French sentent (to smell) is translated
into the German gefühlt.

###  4.4 Evaluation Metrics

As there no ground-truth references exist for the adversarial example task,
prior work has used a number of reference-free metrics to assess each method
(e.g. Xu et al. 2021; Garg and Ramakrishnan 2020), and we also follow the same
approach.

The first metric we use is the label-flip rate: the proportion of attacked
examples where the victim model changes its predicted label compared to the
original example. The next three are measures of text quality and are only
computed for candidates that flip the label. The first of these is an
assessment of the semantic similarity between the original and adversarial
examples, and the metric chosen is BERTScore F1 (Zhang et al. 2020), using a
multilingual BERT variant as the base model. The second is a proxy assessment
of fluency, for which we use BARTScore (Yuan, Neubig, and Liu 2021) which is a
measurement of the likelihood of a given output in a chosen multilingual BART
model. Finally, the third metric assesses if the generated text is in the same
language as the original. We assess this with the confidence output of a
language detection model444We use a model from the Lingua library
(https://github.com/pemistahl/lingua-py)

Eventually, we have combined these metrics into an overall metric, with the
aim to identify candidates that do not obviously violate any of the
adversarial example principles. We call this metric the Validated Success Rate
(VSR) and define it as the proportion of examples that have successfully
flipped the predicted label while also meeting minimum thresholds on all of
the text quality metrics. To set the similarity threshold, we have used as a
guide the similarity scores achieved from the multilingual paraphraser before
training, which trend around 0.65. We have hence set a threshold of 0.6 to
allow the generative model some ‘wiggle room’ in the text generation. For the
language detection, we have empirically set the threshold to 0.5. For the
fluency threshold, we have explored how two different choices (“tight”, -9,
and “loose”, -11) affect the overall VSR achieved by each method, as shown in
Figure 2.

##  5 Results

The results for each language and for the two different fluency thresholds are
shown in Figure 2. For the proposed approach, we only plot one VSR value per
threshold as a dot in correspondence of query limit = 32, since we use a beam
size of 32 and all the beam outputs are assessed against the victim model. For
the baselines, we plot full curves with maximum number of queries in the range
[16, 4096] (blue: mBAE; red: mCLARE). The larger values correspond to the
looser fluency threshold.

The results show that the performance from the proposed generative model has
been remarkable. It has outperformed the baselines in the low-query range by
large margins, and has remained reasonably competitive for larger numbers of
queries. Of the two baselines, mBAE has reported a higher performance for
lower numbers of queries, and mCLARE for higher numbers. The performance
across languages has been fairly consistent throughout, with the exception of
Arabic, for which all methods have struggled. The baselines did more poorly
over the German datasets, but this discrepancy can be partially explained by
the impact of the document length: the test sets for German are made of
shorter documents, and the token replacements made by the baselines affect the
fluency metric more significantly.

A few, selected qualitative examples of the generated attacks are shown in
Table 1. These examples show that the generative approach has been able to
produce effective adversarial examples by applying richer, sentence-level
transforms. By contrast, the baselines’ transforms have been more limited, and
often not of high linguistic quality.

###  5.1 Sensitivity Analysis

The semantic similarity and language detection coefficients in Equation 1
(respectively, $\alpha_{s}$ and $\alpha_{l}$) control the intensity of their
respective regularisers, and in turn impact the quality of the generated text.
We have qualitatively explored the impact of removing these components from
the proposed approach, with examples displayed in Tables 2 and 3. These
examples confirm that removing the similarity component reduces the
faithfulness to the original example, and that removing the language detection
component allows the generative model to “stray” to generate examples
containing text in other languages.

###  5.2 Language Impact

All methods have struggled with Arabic, and this suggests that specialist
adversarial algorithms may be required, such as Alshemali and Kalita (2019).
As a plausible reason, Arabic is highly inflected and differs significantly in
its morphological and syntactic structure from the other four languages,
making it unlikely that the adversarial attacks could effectively exploit
their cross-lingual knowledge. Moreover, in the case of the proposed approach
the base model, mT5, was originally trained on the mC4 corpus (Xue et al.
2021), which contains far fewer Arabic tokens than for the other languages,
and the same applies to the TaPaCo corpus used for the paraphraser training.
In turn, mCLARE and mBAE use multilingual BERT as the base language model,
which was also trained on less Arabic data than for the other languages and is
known to perform poorly compared to monolingual Arabic BERT variants, such as
AraBERT (Antoun, Baly, and Hajj 2020). Therefore, adversarial performance in
Arabic may be improved by adopting dedicated language models. In addition, the
BERT tokeniser also introduces a significant amount of redundancy in tokens
when tokenising Arabic text (Antoun, Baly, and Hajj 2020), and using a
specialist Arabic segmenter such as Farasa (Darwish and Mubarak 2016) could
improve performance.

##  6 Conclusion

In this paper, we have presented a white-box generative approach for attacking
multilingual text classifiers. The training objective of the proposed approach
is very flexible, and incorporates specialised component models to encourage
fluency, semantic consistency, and language adherence. To connect the
generative model to the component models we have leveraged vocabulary-mapping
matrices that allow using models of any vocabulary and retaining the full
differentiability of the training objective. Experimental results across two
multilingual datasets have confirmed the effectiveness of the proposed
approach, particularly in a low number of queries scenario or when richer,
sentence-level transforms are desired. Future research in this area might aim
to adapt this framework to better target under-resourced or linguistically
diverse languages, or to attack models for other multilingual NLP tasks, such
as machine translation or cross-lingual classification.

##  Appendix A Training Details

To train the generative model to paraphrase, we have used the TaPaCo dataset,
using the splits for the chosen five languages. After filtering out examples
longer than our maximum token limit of 32 tokens, we had the following rough
number of examples per language: ar: 6.5k, de: 125k, en: 158k, es: 85k, fr:
117k. We then preprocessed these examples to create a corpus of ordered
paraphrase pairs (i.e. so that (p1,p2) and (p2,p1) would be two separate
paraphrase pairs). We then oversampled or undersampled the pairs to obtain
100k paraphrase pairs per language, and collated them to form our final
training dataset. For the MARC and TSM datasets, we have removed all training
examples that the victim model classified incorrectly prior to the training
procedure, as they could be said to be adversarial already. We have also
removed examples longer than 32 tokens, since the paraphrase model was not
trained for sequences in this range. For the MARC dataset, we have categorised
1 and 2 star reviews as negative, 4 and 5 star reviews as positive, and
discarded the 3 star reviews. Because this is a large dataset we selected a
language-balanced subset to use as the training, validation and test sets,
choosing 5k samples for the training set and 1k samples each for the
validation and test sets. For the TSM dataset, the approximate sizes are in
Table 4.

Lang | Trn / Val / Test
---|---
ar | 0.6k / 0.1k / 0.2k
de | 1.2k / 0.2k / 0.4k
en | 0.6k / 0.1k / 0.3k
es | 1.0k / 0.1k / 0.3k
fr | 0.9k / 0.1k / 0.4k
all | 4.3k / 0.6k / 1.6k
Table 4: Approximate training/validation/test splits of the TSM dataset by
language.

The hyperparameters used for our experiments are listed in Table 5. As
optimiser, we have used Adafactor555We set the following arguments:
scale_parameter=False, relative_step=False, warmup_init=False) with learning
rate set to 0.0001. The models have been trained on a single NVIDIA A40 GPU
with 48 GB RAM.

During training, we have bucketed examples with similar length to form
training batches, and have passed the batches into the dataloader in random
order.

To select the values for the $\alpha$ parameters, we have first identified
plausible ranges by initially performing a random search over the parameter
space. We have then tuned the parameters manually, using subjective judgement
based on our assessment of the quality of the generated text and the
validation set performance. We have repeated the process for each dataset, as
it is unlikely that a single set of parameters would be ideal for all. The
$\beta$ coefficients have also been set manually, through an iterative process
of examination of preliminary results and generated text samples, adjustment,
and repetition, and a subjective examination of what constitutes an
“acceptable” minimum threshold. The $\gamma$ constants in Section 3.5, which
determine when to stop training, were also set in a similar way. The $\tau$
temperature parameter for the Gumbel-softmax sampling was set to 1.1 and we
have not experimented with other values.

Hyperparameter | Value
---|---
General |
Optimisation algorithm | Adafactor
Learning rate | $1\times 10^{-5}$
Weight decay | 0
Batch size (train) | 5
Max original length | 32
Precision | fp32
Coefficients (Equations 1 and 3) | MARC | TSM dataset
Victim ($\alpha_{v}$) | 15 | 15
Similarity ($\alpha_{s}$) | 20 | 25
Language consistency ($\alpha_{l}$) | 20 | 20
KL ($\alpha_{KL}$) | 1 | 4
Diversity ($\alpha_{d}$) | 0.1 | 10
Victim threshold ($\beta_{v}$) | $1-1/c$
Similarity threshold ($\beta_{s}$) | 0.75
Language consistency threshold ($\beta_{l}$) | 0.0
KL threshold ($\beta_{KL}$) | 2
Gumbel-softmax temperature $\tau$ | 1.1
# Gumbel samples | 5
Early-stopping criteria (Section 3.5) |
Victim score ($\gamma_{v}$) | 7
Similarity score ($\gamma_{s}$) | 0.75
Language consistency score ($\gamma_{l}$) | 1
KL score ($\gamma_{KL}$) | 0.6
Validation frequency | Every 24 batches
Patience | 12
Test-set generation |
Batch size (eval) | 5
# generated sequences ($n$) | 32
# beams | 32
# beam groups | 16
Diversity penalty | 1
Top-p | 0.98
Temperature | 1
Min generated length | $\max(0,l-2-\text{floor}(l/4))$
Max generated length | $l+2$
Table 5: Hyperparameters. $c$ is the number of classes in the dataset and $l$
is the batch length of generated text during evaluation (after padding).

##  Appendix B Limitations

One limitation of the proposed approach is that the experiments have only been
conducted on datasets consisting of relatively short examples. However, since
longer texts can always be subdivided, this limitation may not be very
significant. A second limitation is that the approach has only been tested on
five languages, and we do not know how it would perform in more diverse
languages, such as Japanese or Chinese. A third limitation is that the method
requires use of a GPU with a substantial amount of memory, in order to be able
to store all the component models during training. The use of parameter-
efficient fine-tuning methods, such as LoRA (Hu et al. 2022) may be able to
mitigate this issue.

## References

  * Alshemali and Kalita (2019) Alshemali, B.; and Kalita, J. 2019.  Adversarial Examples in Arabic.  In _2019 International Conference on Computational Science and Computational Intelligence (CSCI)_ , 371–376.
  * Antoun, Baly, and Hajj (2020) Antoun, W.; Baly, F.; and Hajj, H. 2020.  AraBERT: Transformer-based Model for Arabic Language Understanding.  In _LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020_ , 9.
  * Barbieri, Espinosa Anke, and Camacho-Collados (2022) Barbieri, F.; Espinosa Anke, L.; and Camacho-Collados, J. 2022.  XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , 258–266. Marseille, France: European Language Resources Association.
  * Cer et al. (2018) Cer, D.; Yang, Y.; Kong, S.-y.; Hua, N.; Limtiaco, N.; St. John, R.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; Strope, B.; and Kurzweil, R. 2018.  Universal Sentence Encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 169–174. Brussels, Belgium: Association for Computational Linguistics.
  * Chang et al. (2023) Chang, G.; Gao, H.; Yao, Z.; and Xiong, H. 2023.  TextGuise: Adaptive adversarial example attacks on text classification model.  _Neurocomputing_ , 529: 190–203.
  * Darwish and Mubarak (2016) Darwish, K.; and Mubarak, H. 2016.  Farasa: A New Fast and Accurate Arabic Word Segmenter.  In Calzolari, N.; Choukri, K.; Declerck, T.; Goggi, S.; Grobelnik, M.; Maegaard, B.; Mariani, J.; Mazo, H.; Moreno, A.; Odijk, J.; and Piperidis, S., eds., _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , 1070–1074. Portorož, Slovenia: European Language Resources Association (ELRA).
  * Das et al. (2021) Das, S. D.; Basak, A.; Mandal, S.; and Das, D. 2021.  AdvCodeMix: Adversarial Attack on Code-Mixed Data.  _5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Garg and Ramakrishnan (2020) Garg, S.; and Ramakrishnan, G. 2020.  BAE: BERT-based Adversarial Examples for Text Classification.  In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , 6174–6181. Association for Computational Linguistics.
  * Guo et al. (2021) Guo, C.; Sablayrolles, A.; Jégou, H.; and Kiela, D. 2021.  Gradient-based Adversarial Attacks against Text Transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 5747–5757. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Hu et al. (2022) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Jang, Gu, and Poole (2017) Jang, E.; Gu, S.; and Poole, B. 2017.  Categorical Reparameterization with Gumbel-Softmax.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Jin et al. (2020) Jin, D.; Jin, Z.; Zhou, J.; and Szolovits, P. 2020.  Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34: 8018–8025.
  * Keung et al. (2020) Keung, P.; Lu, Y.; Szarvas, G.; and Smith, N. A. 2020.  The Multilingual Amazon Reviews Corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Kudo and Richardson (2018) Kudo, T.; and Richardson, J. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 66–71. Brussels, Belgium: Association for Computational Linguistics.
  * Lee et al. (2022) Lee, D.; Moon, S.; Lee, J.; and Song, H. O. 2022.  Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimization.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , 12478–12497. PMLR.
  * Li et al. (2021) Li, D.; Zhang, Y.; Peng, H.; Chen, L.; Brockett, C.; Sun, M.-T.; and Dolan, B. 2021.  Contextualized Perturbation for Textual Adversarial Attack.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5053–5069. Online: Association for Computational Linguistics.
  * Li et al. (2020) Li, L.; Ma, R.; Guo, Q.; Xue, X.; and Qiu, X. 2020.  BERT-ATTACK: Adversarial Attack Against BERT Using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 6193–6202.
  * Liu, Cai, and Qi (2023) Liu, H.; Cai, C.; and Qi, Y. 2023.  Expanding Scope: Adapting English Adversarial Attacks to Chinese.  In Ovalle, A.; Chang, K.-W.; Mehrabi, N.; Pruksachatkun, Y.; Galystan, A.; Dhamala, J.; Verma, A.; Cao, T.; Kumar, A.; and Gupta, R., eds., _Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023)_ , 276–286. Toronto, Canada: Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rosenthal, Bornea, and Sil (2021) Rosenthal, S.; Bornea, M. A.; and Sil, A. 2021.  Are Multilingual BERT models robust? A Case Study on Adversarial Attacks for Multilingual Question Answering.  _ArXiv_ , abs/2104.07646.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Scherrer (2020) Scherrer, Y. 2020.  TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages.
  * Song et al. (2021) Song, L.; Yu, X.; Peng, H.-T.; and Narasimhan, K. 2021.  Universal Adversarial Attacks with Natural Triggers for Text Classification.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 3724–3733. Online: Association for Computational Linguistics.
  * Tan and Joty (2021) Tan, S.; and Joty, S. 2021.  Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots.  In Toutanova, K.; Rumshisky, A.; Zettlemoyer, L.; Hakkani-Tur, D.; Beltagy, I.; Bethard, S.; Cotterell, R.; Chakraborty, T.; and Zhou, Y., eds., _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 3596–3616. Online: Association for Computational Linguistics.
  * Vijayakumar et al. (2018) Vijayakumar, A.; Cogswell, M.; Selvaraju, R.; Sun, Q.; Lee, S.; Crandall, D.; and Batra, D. 2018.  Diverse Beam Search for Improved Description of Complex Scenes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Vijayakumar et al. (2016) Vijayakumar, A. K.; Cogswell, M.; Selvaraju, R. R.; Sun, Q.; Lee, S.; Crandall, D. J.; and Batra, D. 2016.  Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models.  _CoRR_ , abs/1610.02424.
  * Wang et al. (2022) Wang, B.; Xu, C.; Liu, X.; Cheng, Y.; and Li, B. 2022.  SemAttack: Natural Textual Attacks via Different Semantic Spaces.  In Carpuat, M.; de Marneffe, M.-C.; and Meza Ruiz, I. V., eds., _Findings of the Association for Computational Linguistics: NAACL 2022_ , 176–205. Seattle, United States: Association for Computational Linguistics.
  * Wang et al. (2020) Wang, T.; Wang, X.; Qin, Y.; Packer, B.; Li, K.; Chen, J.; Beutel, A.; and Chi, E. 2020.  CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 5141–5146. Online: Association for Computational Linguistics.
  * Wu et al. (2016) Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; Klingner, J.; Shah, A.; Johnson, M.; Liu, X.; Kaiser, L.; Gouws, S.; Kato, Y.; Kudo, T.; Kazawa, H.; Stevens, K.; Kurian, G.; Patil, N.; Wang, W.; Young, C.; Smith, J.; Riesa, J.; Rudnick, A.; Vinyals, O.; Corrado, G.; Hughes, M.; and Dean, J. 2016.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  _CoRR_ , abs/1609.08144.
  * Xu et al. (2021) Xu, Y.; Zhong, X.; Jimeno Yepes, A.; and Lau, J. H. 2021.  Grey-box Adversarial Attack And Defence For Sentiment Classification.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 4078–4087. Online: Association for Computational Linguistics.
  * Xue et al. (2021) Xue, L.; Constant, N.; Roberts, A.; Kale, M.; Al-Rfou, R.; Siddhant, A.; Barua, A.; and Raffel, C. 2021.  mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 483–498. Online: Association for Computational Linguistics.
  * Yang et al. (2020) Yang, Y.; Cer, D.; Ahmad, A.; Guo, M.; Law, J.; Constant, N.; Hernandez Abrego, G.; Yuan, S.; Tar, C.; Sung, Y.-h.; Strope, B.; and Kurzweil, R. 2020.  Multilingual Universal Sentence Encoder for Semantic Retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 87–94. Online: Association for Computational Linguistics.
  * Yoo et al. (2020) Yoo, J. Y.; Morris, J.; Lifland, E.; and Qi, Y. 2020.  Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , 323–332. Online: Association for Computational Linguistics.
  * Yuan et al. (2023) Yuan, L.; Zhang, Y.; Chen, Y.; and Wei, W. 2023.  Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework.  In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., _Findings of the Association for Computational Linguistics: ACL 2023_ , 7132–7146. Toronto, Canada: Association for Computational Linguistics.
  * Yuan, Neubig, and Liu (2021) Yuan, W.; Neubig, G.; and Liu, P. 2021.  BARTScore: Evaluating Generated Text as Text Generation.  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems_ , volume 34, 27263–27277. Curran Associates, Inc.
  * Zhang et al. (2020) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020.  BERTScore: Evaluating Text Generation with BERT.  In _International Conference on Learning Representations_.
  * Zhu, Zhao, and Wu (2023) Zhu, H.; Zhao, Q.; and Wu, Y. 2023.  BeamAttack: Generating High-Quality Textual Adversarial Examples Through Beam Search And Mixed Semantic Spaces.  In _Advances in Knowledge Discovery and Data Mining: 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2023, Osaka, Japan, May 25–28, 2023, Proceedings, Part II_ , 454–465. Berlin, Heidelberg: Springer-Verlag.  ISBN 978-3-031-33376-7.