# Game-Theoretic Unlearnable Example Generator

Shuang Liu1, 2, Yihan Wang1, 2, Xiao-Shan Gao1, 2 Corresponding author.

###### Abstract

Unlearnable example attacks are data poisoning attacks aiming to degrade the
clean test accuracy of deep learning by adding imperceptible perturbations to
the training samples, which can be formulated as a bi-level optimization
problem. However, directly solving this optimization problem is intractable
for deep neural networks. In this paper, we investigate unlearnable example
attacks from a game-theoretic perspective, by formulating the attack as a
nonzero sum Stackelberg game. First, the existence of game equilibria is
proved under the normal setting and the adversarial training setting. It is
shown that the game equilibrium gives the most powerful poison attack in that
the victim has the lowest test accuracy among all networks within the same
hypothesis space, when certain loss functions are used. Second, we propose a
novel attack method, called the Game Unlearnable Example (GUE), which has
three main gradients. (1) The poisons are obtained by directly solving the
equilibrium of the Stackelberg game with a first-order algorithm. (2) We
employ an autoencoder-like generative network model as the poison attacker.
(3) A novel payoff function is introduced to evaluate the performance of the
poison. Comprehensive experiments demonstrate that GUE can effectively poison
the model in various scenarios. Furthermore, the GUE still works by using a
relatively small percentage of the training data to train the generator, and
the poison generator can generalize to unseen data well. Our implementation
code can be found at https://github.com/hong-xian/gue.

## Introduction

Deep learning has achieved remarkable success in various fields, including
computer vision (He et al. 2016), natural language processing, and large
language models (Brown et al. 2020), where the acquisition of a substantial
amount of training data is typically necessary. However, in practical
scenarios, there exists a potential issue of collecting unauthorized private
data from the Internet to train these models. This raises significant privacy
concerns and underscores the need to address how to protect personal data from
exploitation.

To prevent unauthorized use of private data, several poisoning methods (Feng,
Cai, and Zhou 2019; Huang et al. 2021; Fowl et al. 2021; Yuan and Wu 2021; Tao
et al. 2021; Yu et al. 2022; van Vlijmen et al. 2022; Sandoval-Segura et al.
2022b) have been proposed. These methods involve adding imperceptible
perturbations to the training samples, thereby poisoning the models and
leading to poor performance on clean test data, thus making the training data
unlearnable. However, it is generally agreed that the above methods are
vulnerable to adversarial training. In response, novel approaches such as (Fu
et al. 2022; Wen et al. 2023) have been proposed to generate robust
unlearnable examples that can withstand adversarial training. Commonly
referred to as "unlearnable examples attack" or "availability attack", these
poison attacks have attracted significant attention.

In previous works (Feng, Cai, and Zhou 2019; Tao et al. 2021; Yu et al. 2022),
unlearnable example attacks were usually formulated as a bi-level optimization
problem. But directly solving the optimization problem is intractable, and
Feng, Cai, and Zhou (2019) used an alternative update and Yuan and Wu (2021)
used Neural Tangent Kernels (Jacot, Gabriel, and Hongler 2018) to
approximately optimize the bi-level objective. Yu et al. (2022) designed
poison perturbations that are easily learned by the model as “shortcuts” to
prevent learning the information from the real data.

In this paper, we formulate the unlearnable example attack as a non-zero sum
Stackelberg game, in which the attacker, as the leader, crafts poison
perturbation on the training data with the aim of decreasing the test
accuracy, while the classifier, as the follower, optimizes the network
parameters on the poisoned training dataset. We prove the existence of
Stackelberg equilibria in both the normal and adversarial training settings.
Furthermore, we propose a novel and effective approach, namely Game
Unlearnable Example (GUE), in which we directly compute the Stackelberg game
equilibrium using a first-order algorithm (Liu et al. 2022) and select an
appropriate payoff function for the poison attacker. It is noteworthy that our
GUE is not only applicable to standard natural training but can also extend to
adversarial training.

Existing unlearnable example attacks typically require modifying the entire
training dataset. However, in practical scenarios, the continuous influx of
new clean data over time can render the unlearnable effect almost negligible.
To mitigate this problem, when there is a continuous stream of new data, the
user must go through the generation process for the whole dataset repeatedly,
which is time-consuming. In order to overcome this problem, we use an
autoencoder-like generator as the attacker, and show that a well-trained
attacker can easily generalize to unseen data well. The generator can be used
to perform a simple forward propagation on the incoming data to generate the
corresponding poison perturbation. Figure 1 illustrates the pipeline of the
GUE attack.

Figure 1: An illustration of GUE attack, where the trained generator can
generalize to unseen data well.

In summary, our work has three main contributions:

  * •

The unlearnable example attacks are formulated as a Stackelberg game and the
existence of game equilibria under several useful settings are proved. It is
further shown that the game equilibrium gives the most powerful poison attack
in that the victim has the lowest test accuracy among all networks within the
same hypothesis space when certain loss functions are used.

  * •

We propose a new unlearnable example attack, called GUE, which, to the best of
our knowledge, is the first poison method directly based on computing the game
equilibria. Furthermore, by using an autoencoder-like generator as the
attacker, poison perturbations for new data can be generated with simple
forward propagation.

  * •

Extensive experiments are used to demonstrate the effectiveness and
generalizability of GUE in different scenarios. In particular, it is shown
that GUE still works by using a relatively small percentage of the training
data to train the generator.

## Related Works

Our GUE attack is intimately related to unlearnable example attacks, the
Stackelberg game, and bi-level optimization problems. We first provide a brief
overview of the developmental trajectory about unlearnable example attacks and
explain their relationship with our approach.

Unlearnable example attacks are data poisoning methods with addictive bounded
perturbations that prevent unauthorized model training. Error-minimizing noise
(EM) (Huang et al. 2021) generates an imperceptible perturbation by solving a
Min-Min optimization. Hypocritical perturbations (HYP) (Tao et al. 2021)
follows a similar idea but directly uses a pre-trained model rather than the
above Min-Min optimization. Error maximizing noise (TAP) (Fowl et al. 2021)
generates adversarial examples as poisoned data. NTGA (Yuan and Wu 2021)
generates poison perturbations in an ensemble of neural networks modeled with
neural tangent kernels. There exist other methods that do not involve
optimization. LSP (Yu et al. 2022) synthesizes linearly separable
perturbations as attacks, and AR (Sandoval-Segura et al. 2022b) introduces a
generic perturbation that can be utilized in various datasets and
architectures.

However, the above methods cannot poison the adversarial trained model, and
several novel methods were proposed to mitigate this problem. REM (Fu et al.
2022) generates a stronger perturbation by solving a Min-Min-Max three-level
optimization problem. INF (Wen et al. 2023) generates a poison perturbation
based on the induction of indiscriminate features between different classes to
poison an adversarially trained model under a larger adversarial budget.

Lu, Kamath, and Yu (2022) formulated the traditional data poison attacks as a
Stackelberg game and used total gradient descent to solve the game, while they
considered adding a small portion of the poisoned data to the training set
rather than modifying the training samples. And they did not consider the
existence of game equilibrium.

The works most relevant to ours are DeepConfuse (Feng, Cai, and Zhou 2019) and
ShortcutGen(van Vlijmen et al. 2022). Feng, Cai, and Zhou (2019) formulated
the unlearnable examples attack as a bi-level optimization problem, and
optimized the bi-level problem by decoupling the alternating update procedure.
However, this method cannot guarantee the convergence theoretically and
requires multiple rounds of optimization, which is very time-consuming. An
autoencoder-like generator was also used as the attacker in DeepConfuse, but
the generalizability of the generator was not discussed. van Vlijmen et al.
(2022) used a static randomly initialized discriminator to train a poison
generator, in order to encourage the generator to learn spurious shortcuts in
the Min-Min framework (Huang et al. 2021). It is based on their conjecture
that a randomly initialized discriminator provides a noisy mapping between
images and labels and thus lacks theoretical guarantee.

## Unlearnable Example Game

We formulate the unlearnable example attack as a Stackelberg game, called
_unlearnable example game_ , denoted as $\mathcal{G}$. Then, we prove the
existence of its equilibrium in the general case, which is extended to various
attacking scenarios.

### Unified Game Framework

#### Settings.

The game $\mathcal{G}$ has two participants: a poison attacker and a victim
classifier. Let $\mathcal{S}$ be a finitely sampled clean training dataset
from a data distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$,
where $\mathcal{X}\subset\mathbb{R}^{n}$ is the dataset and
$\mathcal{Y}=[K]=\\{i\\}_{i=1}^{K}$ is the label set for $K\in\mathbb{N}_{+}$.
The attacker modifies $x$ in each sample-label pair $(x,y)\in\mathcal{S}$ by
an imperceptible perturbation $\mathcal{A}(x,y)$ such that

$\displaystyle||\mathcal{A}(x,y)||_{\infty}\leq\epsilon.$

In our method proposed in Algorithm 1, $\mathcal{A}$ will be specified as an
encoder-decoder generator. Here, it can be simply regarded as a map
$A(x,y):\mathcal{S}\to\mathbb{R}^{n}$, where $n$ is the dimension of the data.
The victim classifier only has access to the poisoned training set on which a
classifier $f$ with parameters $\theta$ will be trained. The attacker’s goal
is to destroy the performance of $f_{\theta}$. We denote the loss function
used in classifier training by $\mathcal{L}_{c}$ and the loss function used in
the evaluation of attack performance by $\mathcal{L}_{a}$.

#### Game model.

In the presence of a poisoning $\mathcal{A}$ introduced by the game leader,
the payoff function of the victim is the empirical risk in the training
process:

$\displaystyle\mathcal{J}_{c}(\mathcal{A},\theta)$
$\displaystyle\coloneqq\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{S}}\big{[}\mathcal{L}_{c}(x+\mathcal{A}(x,y),y;\theta)\big{]}.$

As the follower of the game, the victim should choose one of the _best
responses_ :

$\displaystyle\theta^{*}\in\text{BR}(\mathcal{A})\coloneqq\operatorname*{arg\,inf}_{\theta^{\prime}}\mathcal{J}_{c}(\mathcal{A},\theta^{\prime}).$

Due to the fact that neural network training usually reaches local optima, we
consider the _$\eta$ -approximately best responses_:

$\displaystyle\theta^{*}\in\text{BR}_{\eta}(\mathcal{A})\coloneqq\\{\theta|\mathcal{J}_{c}(\mathcal{A},\theta)<\inf_{\theta^{\prime}}\mathcal{J}_{c}(\mathcal{A},\theta^{\prime})+\eta\\}.$

The payoff function of the attacker is the negative value of population risk:

$\displaystyle\mathcal{J}_{a}(\mathcal{A},\theta)\coloneqq\sup_{\theta\in\text{BR}_{\eta}(\mathcal{A})}\\{-\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{D}}\big{[}\mathcal{L}_{a}(x,y;\theta)\big{]}\\}.$
(1)

The attacker should choose a _Stackelberg strategy_

$\displaystyle\mathcal{A}^{*}\in\operatorname*{arg\,inf}_{\mathcal{A}}\mathcal{J}_{a}(\mathcal{A},\theta).$
(2)

Though the Stackelberg strategy is not ensured unique, the Stackelberg cost is
unique. For a Stackelberg strategy $\mathcal{A}^{*}$, each
$\theta^{*}\in\text{BR}_{\eta}(\mathcal{A}^{*})$ is an approximate optimal
strategy for the victim with respect to $\mathcal{A}^{*}$. We call the action
profile $(\mathcal{A}^{*},\theta^{*})$ a Stackelberg equilibrium without
distinction.

### Equilibrium Existence

One of the primary concern for the game $\mathcal{G}$ is whether this game
possesses an equilibrium. We will prove the existence of the equilibrium,
consequently providing a well-defined target for learning. Next section will
delve into the methods for solving this game.

Assume that the parameters $\theta$ of the victim classifier are restricted in
a feasible strategy space $\Theta\subset\mathbb{R}^{N}$, where $N$ is the
number of parameters. The following common assumptions are required for the
existence of a Stackelberg equilibrium.

######  Assumption 1.

The strategy space $\Theta$ is compact, for instance $\Theta=[-E,E]^{N}$ for
$E\in\mathbb{R}_{+}$.

######  Assumption 2.

The loss functions $\mathcal{L}_{a}(x,y;\theta)$ and
$\mathcal{L}_{c}(x,y;\theta)$ are continuous in $x$ and $\theta$.

######  Theorem 3.

Under Assumptions 1 and 2, the unlearnable example game $\mathcal{G}$ has a
Stackelberg equilibrium $(\mathcal{A}^{*},\theta^{*})$.

###### Proof sketch..

Let $\Gamma$ be the feasible space for the perturbations on the training set,
that is, $\mathcal{A}(\mathcal{S})\subset\Gamma$. With a slight abuse of
notation, we denote $\mathcal{A}\in\Gamma$.

By Assumptions 1 and 2, $\mathcal{J}_{c}(\mathcal{A},\theta)$ is continuous,
$\Theta$ and $\Gamma$ are compact. Then, for any $\mathcal{A}$,
$\text{BR}_{\eta}(\mathcal{A})$ is non-empty. Let
$\mathcal{J}(\theta)\coloneqq-\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{D}}\big{[}\mathcal{L}_{a}(x,y;\theta)\big{]}$.
Thus the existence of Stackelberg equilibrium is equivalent to the existence
of $\mathcal{A}^{*}\in\Gamma$ such that

$\displaystyle\sup_{\theta\in\text{BR}_{\eta}(\mathcal{A}^{*})}\mathcal{J}(\theta)=\inf_{\mathcal{A}}\sup_{\theta\in\text{BR}_{\eta}(\mathcal{A})}\mathcal{J}(\theta).$

Since $\mathcal{J}(\theta)$ is continuous in $\Theta$, the marginal function

$\displaystyle
V(\mathcal{A})\coloneqq\sup\limits_{\theta\in\text{BR}_{\eta}(\mathcal{A})}\mathcal{J}(\theta)$

is lower semi-continuous. Then $V(\mathcal{A})$ can attain the minimum as
$\Gamma$ is compact, that is, there exists an $\mathcal{A}^{*}$ such that

$\displaystyle
V(\mathcal{A}^{*})=\inf_{\mathcal{A}\in\Gamma}V(\mathcal{A})=\sup_{\theta\in\text{BR}_{\eta}(\mathcal{A}^{*})}\mathcal{J}(\theta).$

Since $\text{BR}_{\eta}(\mathcal{A})$ is not empty, there exists a
$\theta^{*}\in\text{BR}_{\eta}(\mathcal{A}^{*})$, and the theorem is proved.
Details of the proof are given in subsection Proof of Theorem 3. ∎

### Various Attacking Scenarios

We have proved the existence of Stackelberg equilibrium with respect to
general loss functions $\mathcal{L}_{c}$ and $\mathcal{L}_{a}$. Next, we will
consider specific loss functions for different scenarios below. As long as
$\mathcal{L}_{c}$ and $\mathcal{L}_{a}$ satisfy Assumption 2, the specified
unlearnable example game has an equilibrium.

#### Standard setting.

The most common attack scenario for classification tasks leverages cross-
entropy loss as both $\mathcal{L}_{c}$ and $\mathcal{L}_{a}$. In this case,
the attacker aims at reducing standard accuracy on the original data
distribution while the victim trains classifier $f_{\theta}$ to improve
standard accuracy on the poisoned training set. We have the following
corollary as a direct consequence of Theorem 3.

######  Corollary 4.

Let $\mathcal{L}_{c}=\mathcal{L}_{a}=\mathcal{L}_{ce}$, then the unlearnable
example game $\mathcal{G}$ has a Stackelberg equilibrium.

In practice, solving this game also requires taking into account the
effectiveness of the algorithm. Searching for a Stackelberg strategy of the
attacker refers to minimizing its payoff function in Equation (1) involving
$\mathcal{L}_{ce}$. However, the cross-entropy loss has no upper bound, and
thus naturally lacks a criterion for convergence of such an optimization.
Furthermore, it results in a gradient explosion, as will be discussed in the
experimental section.

To tackle this issue, we instead leverage a surrogate loss function which is
upper-bounded and is equivalent to cross-entropy loss for solving the game.

#### Surrogate loss.

We propose a novel loss function for the attacker to measure the poison
performance:

$\mathcal{L}_{sur}(x,y;\theta)\coloneqq-\max_{k\in[K]\setminus\\{y\\}}\mathcal{L}_{ce}(x,k;\theta).$
(3)

Intuitively, maximizing $\mathcal{L}_{sur}$ means minimizing the cross-entropy
loss of $x$ with respect to all other labels instead of $y$, leading to the
misclassification of the model trained on the poisoned training set. When
maximizing $\mathcal{L}_{ce}$, it often forms a deep valley on the true label
in the label confidence distribution. On the other hand, when maximizing
$\mathcal{L}_{sur}$, in addition to forming a deep valley on the true label,
it also makes the confidence levels of other labels uniform.

The following properties of surrogate loss allow our method to effectively
solve the game. Moreover, an equilibrium still exists for surrogate loss.

######  Proposition 5.

Assume that the data have $K$ classes.

  1. 1.

$\mathcal{L}_{sur}$ is upper-bounded:

$\displaystyle\mathcal{L}_{sur}(x,y;\theta)\leq-\log(K-1).$

  2. 2.

$\mathcal{L}_{ce}$ grows as $\mathcal{L}_{sur}$ grows:

$\displaystyle\mathcal{L}_{ce}(x,y;\theta)\geq-\log(1-(K-1)e^{\mathcal{L}_{sur}(x,y;\theta)}).$

######  Corollary 6.

Let $\mathcal{L}_{c}=\mathcal{L}_{ce}$ and
$\mathcal{L}_{a}=\mathcal{L}_{sur}$. We denote this game as
$\mathcal{G}_{ue}$. Then this unlearnable example game $\mathcal{G}_{ue}$ has
a Stackelberg equilibrium $(\mathcal{A}_{ue}^{*},\theta_{ue}^{*})$.

#### Adversarial setting.

Recent studies (Tao et al. 2021) have suggested that adversarial training can
mitigate the impact of unlearnable example attacks. Then poisoning approaches
(Fu et al. 2022; Wen et al. 2023) were proposed to work for adversarially
trained models.

Our unified game framework also includes the adversarial scenario in which the
victim is aware of the potentially poisoned training set and decides to deploy
adversarial training and the attacker still wants to prevent the victim from
obtaining an available model.

We need only to modify the training loss $\mathcal{L}_{c}$ to adversarial
settings. Here we consider the two widely-used adversarial losses:

  * •

Adversarial Loss (Madry et al. 2017):

$\displaystyle\mathcal{L}_{adv}(x,y;\theta)\coloneqq\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}\mathcal{L}_{ce}(x+\mu,y;\theta)$

where $\epsilon_{d}$ is the adversarial training radius.

  * •

TRADES Loss (Zhang et al. 2019):

$\displaystyle\mathcal{L}_{tra}(x,y;\theta)\coloneqq$
$\displaystyle\frac{1}{\lambda}\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}\text{KL}(f_{\theta}(x)||f_{\theta}(x+\mu))$
$\displaystyle+\mathcal{L}_{ce}(x,y;\theta).$

The following lemma states that $L_{adv}$ and $L_{tra}$ satisfy Assumption 2.

######  Lemma 7.

$\mathcal{L}_{adv}(x,y;\theta)$ and $\mathcal{L}_{tra}(x,y;\theta)$ are
continuous.

As a consequence of Theorem 3, the equilibrium exists in the adversarial
scenario.

######  Corollary 8.

Let $\mathcal{L}_{c}=\mathcal{L}_{adv}$ or $\mathcal{L}_{tra}$, and
$\mathcal{L}_{a}=\mathcal{L}_{sur}$. We denote this game by
$\mathcal{G}_{at}$. Then this unlearnable example game $\mathcal{G}_{at}$ has
a Stackelberg equilibrium $(\mathcal{A}_{at}^{*},\theta_{at}^{*})$.

It implies that there exists a poisoning attack which can degrade the
classification performance of a model that obtains approximately optimal
parameters through adversarial training on the poisoned training set. We
mainly consider the case with $\mathcal{L}_{c}=\mathcal{L}_{tra}$ in the
following paper.

######  Remark 9.

Similar to (Gao, Liu, and Yu 2022), by using the loss function
$\mathcal{L}_{cw}(f_{\theta}(x),y)=\max_{l\neq
y}[f_{\theta}(x)]_{l}-[f_{\theta}(x)]_{y}$, we can prove that the
corresponding game equilibrium exists and gives the most powerful poison
attack in that the victim has the lowest test accuracy among all networks
within the hypothesis space, when trained on poisoned training set.

## Unlearnable Example Generator

In the preceding section, we theoretically established the unlearnable example
game framework and demonstrated the existence of equilibrium in various
scenarios.

In this section, by solving the game using a first-order algorithm, we propose
a novel unlearnable example generator that can effectively generalize to
future data added to the training set. By doing so, we aim to reduce the high
dependency of unlearnable examples on the poisoning ratio to better align with
the real-world need for protecting unauthorized data. Moreover, this game-
theoretic approach can extend to the adversarial setting.

### Protection of Unauthorized Data

In the era of big data, countless personal data is uploaded to servers through
various applications, published on the Internet, and can be accessed by others
without restrictions. If these data are obtained without authorization by
malicious individuals and used to train machine learning models, there is a
risk that the user’s personal privacy information being misused. As a means of
protecting unauthorized data, unlearnable examples are used to preprocess
upcoming data before its release, rendering models trained on the processed
data unusable.

#### Challenges.

However, there are some common issues that hinder the deployment and
application of existing unlearnable example methods in the real world.
Challenges include

  * •

Dependency on extremely high poisoning ratio.

  * •

Inefficiency of poison generation.

  * •

Vulnerability to recovery approaches.

First, even a slight decrease in the poisoning ratio can result in a
substantial decrease in unlearnability (Huang et al. 2021). When a poisoned
training set is diluted by a few new clean samples, the protection of the data
will be destroyed. It brings an essential obstacle to the application of
unlearnable examples in the real world. Furthermore, most of the existing
unlearnable example attacks are based on an enormous number of iterative
propagation and back-propagation (Fowl et al. 2021). When it comes to
redeploying poisoning to new data flow, the issue of inefficiency becomes more
prominent. Last but not least, adversarial training can restore the usability
of models trained on training sets polluted by poisons that have not been
specifically designed for adversarial training (Tao et al. 2021). Class-wise
perturbations are easier to recover compared to sample-wise perturbations
(Sandoval-Segura et al. 2022a).

### Generalizable Poison Generator

To mitigate the aforementioned problems, we propose a novel approach that
differs from existing methods. Instead of empirically optimizing perturbations
to create shortcuts for learning, we suggest using a game-theoretic framework
to train an unlearnable example generator that can generalize to unseen data
effectively.

In detail, our approach involves having the attacker in the game utilize a
poison generator $g_{\omega}$ based on an encoder-decoder with parameters
$\omega$. That is, $\mathcal{A}(x,y)=g_{\omega}(x)$ which outputs sample-wise
perturbations. The activation for the final layer is
$\epsilon\cdot\tanh(\cdot)$ to control the perturbation within the poison
radius $\epsilon$, i.e. $||g_{w}(x)||_{\infty}\leq\epsilon$.

Now the classifier aims at minimizing the payoff function:

$\displaystyle\mathcal{J}_{c}(w,\theta)=\mathop{\mathbb{E}}\limits_{(x,y)\sim
S}\big{[}\mathcal{L}_{c}(x+g_{w}(x),y;\theta)\big{]}$

by choosing parameters

$\displaystyle\theta^{*}\in\text{BR}_{\eta}(w)=\\{\theta|\mathcal{J}_{c}(w,\theta)<\inf_{\theta^{\prime}}\mathcal{J}_{c}(w,\theta^{\prime})+\eta\\}.$

The attacker chooses generator parameters $w^{*}$ to minimize the payoff
function:

$\displaystyle\mathcal{J}_{a}(\omega,\theta)\coloneqq\sup_{\theta\in\text{BR}_{\eta}(\omega)}\\{-\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{S}}\big{[}\mathcal{L}_{a}(x,y;\theta)\big{]}\\}.$
(4)

To make it feasible in practical computation, here we consider the loss on
clean training set $\mathcal{S}$ instead of the data distribution
$\mathcal{D}$ in Equation (1).

Once a generator $g_{\omega}$ is well trained on a given training set, it
costs only forward propagation to generate poisons for images in the training
set and potentially for future images that may be added to the training set.
Due to the flexible selection of $\mathcal{L}_{c}$ and $\mathcal{L}_{a}$, the
generator is able to deal with different scenarios, including standard and
adversarial settings.

### Compute the Game Equilibrium

In fact, a Stackelburg game can be simplified to a bi-level optimization
problem:

$\displaystyle\min_{w,\theta}l(w,\theta)$ (5)
$\displaystyle\text{s.t.}\quad\theta$
$\displaystyle\in\operatorname*{arg\,min}_{\theta^{\prime}}h(w,\theta^{\prime}).$

In the context of an unlearnable example game, we have $l=\mathcal{J}_{a}$ and
$h=\mathcal{J}_{c}$. Directly solving a bi-level optimization problem requires
the calculation of the second-order Hessian matrices, which makes it
impractical for common machine learning tasks with high-dimensional inputs and
parameters.

We leverage previously proposed BOME (Liu et al. 2022) and dynamic barrier
gradient descent algorithm (Gong, Liu, and Liu 2021) to efficiently train an
unlearnable example generator, the detailed algorithm is presented in
algorithm 1. BOME is a recently developed first-order bi-level optimization
algorithm that employs a value function approach to transform the bi-level
problem into a single-level constrained optimization problem. Then the single-
level problem is solved by the dynamic barrier gradient descent algorithm,
which has the notable advantage of not requiring the lower-level problem to
possess a unique solution.

#### BOME.

Assume $h(w,\cdot)$ can attain a minimum for each $w$. Problem (5) is
equivalent to the following constrained optimization (even for nonconvex $h$):

$\displaystyle\min_{w,\theta}l(w,\theta)$ $\displaystyle\text{s.t.}\quad
q(w,\theta)$ $\displaystyle\coloneqq h(w,\theta)-h(w,\theta^{*}(w))\leq 0,$

where $h(w,\theta^{*}(w))=\min_{\theta}h(w,\theta)$. In practice, we
approximate $\theta^{*}(w)$ by $\theta^{T}(w)$ which is the $T$ step gradient
descent of $h(w,\cdot)$ over $\theta$ for some step size $\alpha>0$ :

$\theta^{t+1}(w)=\theta^{t}(w)-\alpha\nabla_{\theta}h(w,\theta^{t}(w)).$ (6)

Then we obtain an estimate of $q(w,\theta)$:

$\displaystyle\widehat{q}(w,\theta)=h(w,\theta)-h(w,\theta^{T}(w)).$

#### Dynamic barrier gradient descent.

The method is to iteratively update $(w,\theta)$ with step size $\beta>0$ to
reduce $l$ while controlling the decrease of $q$:

$(w_{k+1},\theta_{k+1})\leftarrow(w_{k},\theta_{k})-\beta(\nabla
l(w_{k},\theta_{k})+\lambda_{k}\nabla\widehat{q}(w_{k},\theta_{k}))$

where

$\displaystyle\lambda_{k}=\max(\frac{\phi_{k}-\left\langle\nabla
l(w_{k},\theta_{k}),\nabla\widehat{q}(w_{k},\theta_{k})\right\rangle}{||\nabla\widehat{q}(w_{k},\theta_{k})||^{2}},0)$

and $\phi_{k}=\rho||\nabla\widehat{q}(w_{k},\theta_{k})||^{2}$ by default with
a hyper-parameter $\rho>0$.

## Experiments

In this section, we conduct comprehensive experiments to validate the
effectiveness of GUE on popular benchmark datasets. Here, we consider not only
standard training but also adversarial training.

### Experiment Settup

We mainly conduct experiments on image classification datasets: CIFAR-10,
CIFAR-100 which have been commonly used in the poisoning literature. We use
ResNet-18 (He et al. 2016) as classifier $f(\theta)$ and U-Net (Ronneberger,
Fischer, and Brox 2015) as poison generator $g_{w}$ during training. If not
explicitly mentioned, we focus on the reasonable setting with poison radius
$\epsilon=8/255$.

We use the test accuracy on clean test set to assess the effectiveness of
unlearnable example attacks; the lower accuracy implies that the attack is
stronger to prevent the model from learning information from the poisoned
training dataset.

Algorithm 1 Training of unlearnable example generator

Input: Training set $\mathcal{S}$, inner step $T$, inner and outer step size
$\alpha,\beta$, batch size $b$, train epochs $e$.
Output: Learned poison Generator $g_{w}$

1: Initialize classifier $f_{\theta}$ and attacker $g_{w}$

2: for $k=1$ to $e$ do

3: Sample a mini-batch $\\{(x_{i},y_{i})\\}_{i=1}^{b}\sim\mathcal{S}$

4: Compute $\theta^{T}(w_{k})$ by $T$ steps gradient descent on
$J_{c}(w_{k},\cdot)$ starting from $\theta_{k}$(like Eq.(6))

5: set
$\widehat{q}(w_{k},\theta_{k})=J_{c}(w_{k},\theta_{k})-J_{c}(w_{k},\theta^{T}(w_{k}))$

6: Update $(w,\theta)$:  $\theta_{k+1}\leftarrow\theta_{k}-\beta(\nabla
J_{a}(\theta_{k})+\lambda_{k}\nabla_{\theta_{k}}\widehat{q}(w_{k},\theta_{k}))$
$w_{k+1}\leftarrow
w_{k}-\beta(\lambda_{k}\nabla_{w_{k}}\widehat{q}(w_{k},\theta_{k}))$ where
$\lambda_{k}=\max(\frac{\phi_{k}-\left\langle\nabla
J(\theta_{k}),\nabla_{\theta_{k}}\widehat{q}(w_{k},\theta_{k})\right\rangle}{||\nabla\hat{q}(w_{k},\theta_{k})||^{2}},0)$
and $\phi_{k}=\rho||\nabla\widehat{q}(w_{k},\theta_{k})||^{2}$. (Set
$\rho=1.5$ and $T=10$ as default)

7: end for

8: return $g_{w}$

### Standard Training

In this subsection, we conduct experiments under standard training setting to
verify the effectiveness of our GUE attack by directly solving the game
$\mathcal{G}_{ue}$ defined in Corollary 6.

We train 50 epochs to generate GUE with algorithm 1, using SGD optimizer with
learning rate 0.01 for classifier $f_{\theta}$ and SGD optimizer with learning
rate 0.1 for attacker $g_{w}$. And use Adam (Kingma and Ba 2014) with learning
rate 0.001 for the inner T steps approximation. And for evaluation of
unlearnable examples, we train a model on poisoned dataset for 100 epochs
using SGD optimizer with an initial learning rate of 0.01 that is decayed by a
factor of 0.1 at the 75-th and 90-th training epochs. The optimizer is set
with momentum 0.9 and weight decay $5\times 10^{-4}$.

#### Compare to different unlearnable example attacks.

We first evaluate the performance of our GUE and different state-of-the-art
poisoning methods under standard training, including DeepConfuse (Feng, Cai,
and Zhou 2019), Unlearnable Examples(EM) (Huang et al. 2021), Targeted
Adversarial Poisoning(TAP) (Fowl et al. 2021) and ShortcutGen (van Vlijmen et
al. 2022). As shown in Table 1, our GUE can substantially decrease the clean
test accuracy and performs better than all methods except EM. Furthermore, as
we can see from Figure 2, our GUE remains unlearnably effective throughout all
training epochs, while all other attacks exhibit an accuracy peak at the
beginning of the training.

Poison method | CIFAR10 | CIFAR100
---|---|---
None(Clean) | 92.36 | 70.59
EM | 10.16 | 1.90
TAP | 20.28 | 15.40
DeepConfuse | 22.74 | 25.73
ShortcutGen | 24.42 | 8.62
GUE | 13.25 | 8.35
Table 1: Test accuracy of ResNet-18 trained on poisoned data from different
unlearnable example attacks on CIFAR. Figure 2: Test accuracy curves of
ResNet-18 trained on poisoned data from different unlearnable example attacks
on CIFAR-10.

#### Different percentages of data to train $g_{w}$ and generalizability.

In practice, it is not always possible that the attacker has access to the
full training dataset, the victim classifier may collect more data to train a
model. It has been observed that other unlearnable example attacks such as EM
or TAP would fail to poison the model when there is a small proportion of
clean data, such as $10\%$ from the training dataset. Thus, when there are
more clean data, these attacks require regenerating corresponding unlearnable
examples on the entire training data after adding new clean data. This
motivates us to examine the generalizability of the poison generator $g_{w}$
trained on only a proportion of randomly selected training examples, and when
there are more clean data, we can use $g_{w}$ to generate unlearnable examples
for these data.

Figure 3: Test accuracy of ResNet-18 trained on poisoned CIFAR-10, where the
poison generator is trained on different percentage of training data.

We use different percentages of training data to train the poison generator,
then use the corresponding generator to generate unlearnable examples on the
entire training dataset. The results are shown in Figures 3 and 4. We can
observe that our GUE’s generalizability is much better than DeepConfuse and
ShortcutGen. Specifically, our GUE is still effective in degrading the test
accuracy to about $20\%$ when we only use $60\%$ of the training data in
CIFAR10. This indicates that the GUE-trained poison generator can generalize
to unseen data well.

Figure 4: Test accuracy of ResNet-18 trained on poisoned CIFAR-100, where the
poison generator is trained on different percentage of train data.

#### Transferability on different model architectures.

We generate a GUE with fixed classifier structure as ResNet-18. For this
reason, a natural question to ask is whether the poison generator trained with
one model architecture is still effective when the victim classifier adopts a
different architecture. Table 2 shows that the poisoning effects of our GUE
optimized against a ResNet-18 classifier can be transferred to other model
architectures. (To save space, we denote DeepConfuse and ShorcutGen as DC and
SG.) Specifically, we can observe that the transferability of our GUE is more
stable than that of DeepConfuse and ShortcutGen. Our GUE is capable of
reducing the test accuracy to $12.97\%\sim 14.68\%$ across various model
architectures.

Poison method | Clean | GUE | DC | SG
---|---|---|---|---
ResNet-18 | 92.36 | 13.25 | 22.74 | 24.42
VGG16 | 91.18 | 13.72 | 25.35 | 12.32
ResNet-50 | 92.14 | 12.97 | 20.56 | 17.35
DenseNet-121 | 92.10 | 13.71 | 21.44 | 16.59
WRN-28-10 | 92.86 | 14.68 | 29.02 | 25.09
Table 2: Transferability of GUE from ResNet-18 to other model architectures on
CIFAR-10.

#### Effects of defenses against GUE.

We have demonstrated the efficacy of our GUE in the setting where the victim
classifier only uses standard training. However, the victim classifier could
employ several defenses proposed against poison attacks. Thus, we test the
effectiveness of several popular defenses against our GUE.

Here, we firstly study whether data augmentations can mitigate our GUE.
Following previous work, we test a diverse set of data augmentations,
including Mixup (Zhang et al. 2017), Cutout (DeVries and Taylor 2017), and
Cutmix (Yun et al. 2019). Since adversarial training is widely considered as
an effective defense against unlearnable examples attacks, we also test
adversarial training with adversarial radius $\epsilon_{d}=2/255$ and $4/255$.

Table 3 shows that all data augmentation methods fail to mitigate our GUE.
Adversarial training can counteract the poison effect, but our GUE is still
effective when the adversarial radius is small $\epsilon_{d}=2/255$. It is
more effective than other existing methods, such as EM, TAP, both of which
achieve test accuracy exceeding $70\%$.

Defenses | Clean test accuracy
---|---
Baseline(Clean) | 92.36
Mixup | 11.83
Cutout | 14.71
Cutmix | 19.84
Adv training($\epsilon_{d}=2/255$) | 22.55
Adv training($\epsilon_{d}=4/255$) | 76.96
Table 3: Evaluating GUE against different defenses.

### Adversarial Training

We find that our GUE cannot defend against adversarial training for a large
adversarial radius in the preceding subsection. To solve this problem, we
consider solving the game $\mathcal{G}_{at}$ in which the victim classifier
also adopts adversarial training. And we denote the attacks as GUE-
AT($\epsilon,\epsilon_{d}$) where $\epsilon$ is the poison radius and
$\epsilon_{d}$ is the adversarial radius in the game $\mathcal{G}_{at}$.

We train 150 epochs to generate GUE-AT with algorithm 1, the optimizers are
set as the standard training. We set $\lambda=1$ in trade-off loss, the inner
maximization (that is, generation of adversarial examples) of the adversarial
training is solved by 10-step PGD with a step size of $\epsilon_{d}/4$. For
adversarial training evaluation experiments, all training settings are the
same as the standard training except that the initial learning rate is changed
to 0.1.

#### Different adversarial training perturbation radii.

We train models using different adversarial training perturbation radii on
these unlearnable examples. The adversarial training radius $\epsilon_{d}$
ranges from $0/255$ to $4/255$. It is important to note that when
$\epsilon_{d}=0$, the models are trained using the standard training method.
Table 4 reports the accuracies of the trained models under different
unlearnable example attacks.

We can see that our GUE-AT outperforms the SOTA method REM over
$7.48\%-9.71\%$ when adversarial radius $\epsilon_{d}=2/255$. For a larger
adversarial radius, our GUE-AT is also effective. In total, these experiments
show that our method can be applied to different adversarial training
perturbation radii while keeping a significant poison effect.

Poison method | $\epsilon_{d}=0$ | $\epsilon_{d}=2/255$ | $\epsilon_{d}=4/255$
---|---|---|---
None(Clean) | 92.36 | 92.41 | 89.82
INF | 88.78 | 80.91 | 77.96
REM(8-2) | 21.14 | 29.75 | 74.20
GUE-AT(8-2) | 16.11 | 22.27 | 60.11
REM(8-4) | 26.15 | 30.90 | 44.91
GUE-AT(8-4) | 17.86 | 21.19 | 52.79
Table 4: Test accuracy of ResNet-18 adversarially trained with radius
$\epsilon_{d}=0,2/255,4/255$. INF is from (Wen et al. 2023).

### Ablation Studies

In this subsection, we conduct experiments to provide an empirical
understanding of the proposed surrogate loss. As discussed when introducing
the surrogate loss, the cross-entropy loss has no upper bound and does not
provide a criterion for convergence. Therefore, we propose the surrogate loss
$\mathcal{L}_{sur}$, which is upper bounded and equivalent to
$\mathcal{L}_{ce}$. Empirically, we compute the game equilibrium with
$L_{a}=\mathcal{L}_{ce}$ and $\mathcal{L}_{a}=\mathcal{L}_{sur}$, and observe
the gradient of $\mathcal{J}_{a}$ and the poisoned loss
$\mathcal{J}_{c}(w,\theta)$ in each batch of data during training.

Figure 5 (a) shows that $||\nabla_{\theta}\mathcal{J}_{a}||_{2}$ with
$\mathcal{L}_{ce}$ is approximately two orders of magnitude larger than with
$\mathcal{L}_{sur}$, and diverges. A straightforward method to overcome the
explosion of gradients is gradient clipping. Hence, we use gradient clipping
to control the gradient norm $||\nabla_{\theta}\mathcal{J}_{a}||_{2}=10$.
However, in Figure 5(b) it can be seen that, as a necessary condition for
convergence, poisoned loss $\mathcal{J}_{c}(w,\theta)$ cannot converge when we
use $\mathcal{L}_{ce}$ or $\mathcal{L}_{ce}$ with gradient clipping. Only our
proposed loss has guarantee of convergence. This verifies the necessity of our
proposed loss $\mathcal{L}_{sur}$ to compute the equilibrium.

(a) $||\nabla_{\theta}\mathcal{J}_{a}||_{2}$

(b) Poisoned loss $\mathcal{J}_{c}(w,\theta)$

Figure 5: The training curves of $||\nabla_{\theta}\mathcal{J}_{a}||_{2}$ and
$\mathcal{J}_{c}(w,\theta)$ when we use different loss function in
$\mathcal{J}_{a}$ to compute the equilibrium:
$\mathcal{L}_{ce},\mathcal{L}_{sur}$ and $\mathcal{L}_{ce}$ with gradient
clipping. We take $\log_{10}$ of all values for better visualization.

## Proofs

In this section, we give proofs of the theoretical results in the paper and
give a complete statement of Remark 9.

### Proof of Theorem 3

We first introduce some lemmas about the semicontinuity of set-valued maps and
then give the complete proof of the existence of the Stackelberg equilibrium
of unlearnable examples game.

Similar to Lemma 3.2 in (Gao, Liu, and Yu 2022), we can prove the following
properties of payoff functions.

######  Lemma 10.

The payoff functions $J_{c}(\mathcal{A},\theta)$ and $J_{a}(\theta)$ are
continuous.

######  Lemma 11.

Let $X,Y$ be two Hausdorff spaces and $X$ compact, $F:X\times
Y\rightarrow\mathbb{R}$ continuous. Then $\inf\limits_{x\in X}F(x,y)$ is
continuous on $Y$.

###### Proof.

It is equivalent to prove that $\forall\epsilon>0,\exists\delta>0$, for any
$||y_{1}-y_{2}||\leq\delta$, we have $|\inf_{x\in X}F(x,y_{1})-\inf_{x\in
X}F(x,y_{2})|\leq\epsilon$. Since $F(x,y)$ is continuous and $X$ is compact,
there exists an $x^{*}$ for any $y$ such that $F(x^{*},y)=\inf_{x\in
X}F(x,y)$, so there exist $x_{1}^{*},x_{2}^{*}$ correlated with $y_{1},y_{2}$,
respectively.

Since $F(x,y)$ is continuous, $\forall\epsilon>0,\exists\delta(x)>0$, such
that when $||y_{1}-y_{2}||\leq\delta(x)$, we have
$|F(x,y_{1})-F(x,y_{2})|\leq\epsilon$. Hence $\forall\epsilon>0$, there exits
a $\delta(x_{2}^{*})$, such that

$\inf_{x\in X}F(x,y_{1})-\inf_{x\in X}F(x,y_{2})\leq
F(x_{2}^{*},y_{1})-F(x_{2}^{*},y_{2})\leq\epsilon$

and there exits a $\delta(x_{1}^{*})$ such that

$\displaystyle\inf_{x\in X}F(x,y_{1})-\inf_{x\in X}F(x,y_{2})$
$\displaystyle\geq F(x_{1}^{*},y_{1})-F(x_{1}^{*},y_{2})\geq-\epsilon.$

Let $\delta=\min\\{\delta(x_{1}^{*}),\delta(x_{2}^{*})\\}$. Then
$\forall\epsilon>0,\exists\delta>0$, such that for any
$||y_{1}-y_{2}||\leq\delta$, $|\inf_{x\in X}F(x,y_{1})-\inf_{x\in
X}F(x,y_{2})|\leq\epsilon$ holds, that is, $\inf\limits_{x\in X}F(x,y)$ is
continuous on $Y$. ∎

######  Lemma 12.

(Aubin and Ekeland 1984) Let $X,Y$ be two Hausdorff spaces, $G$ a set-valued
map from $Y$ to $X$, and $W$ a real-valued function defined on $X$. Suppose
that $W$ is lower semicontinuous, $G$ is lower semicontinuous on $Y$. Then the
marginal function $V(y)=\sup\limits_{x\in G(y)}W(x)$ is lower semicontinuous
on $Y$.

######  Lemma 13.

Let $X,Y$ be two compact subsets in a metric space, $F:X\times
Y\rightarrow\mathbb{R}$ a continuous function, and $G(y)=\\{x\in
X:F(x,y)<\inf\limits_{z\in X}F(z,y)+\eta\\}$ where $\eta>0$ is a constant.
Then $G(y):Y\rightarrow X$ is lower semicontinuous.

###### Proof.

The fact that $G(y)$ is lower semicontinuous is equivalent to proving for any
$y\in Y$, any sequence $\\{y^{m}\\}_{m=1}^{\infty}$ convergent to $y$, and
$\forall x^{0}\in G(y)$, there exists a sequence $\\{x^{m}\\}_{m=1}^{\infty}$
converging to $x^{0}$ such that $x^{m}\in G(y^{m})$ holds for sufficiently
large $m$.

Since $x^{0}\in G(y)$, we have $F(x^{0},y)<\inf\limits_{x}F(x,y)+\eta$, and
$X$ is compact, so there exists a sequence $\\{x^{m}\\}_{m=1}^{\infty}$ that
converges to $x^{0}$. Because $x^{m}\rightarrow x^{0}$ and $y^{m}\rightarrow
y$, and $F(x,y)$ are continuous, we have
$\lim\limits_{m\rightarrow\infty}F(x^{m},y^{m})=F(x^{0},y)$. According to the
Lemma 11, since $X$ is compact, $\inf\limits_{x\in X}F(x,y)$ is continuous on
$Y$. Then

$\displaystyle\lim\limits_{m\rightarrow\infty}F(x^{m},y^{m})$
$\displaystyle=F(x^{0},y)<\inf\limits_{x}F(x,y)+\eta$
$\displaystyle=\lim\limits_{m\rightarrow\infty}\inf\limits_{x\in
X}F(x,y^{m})+\eta.$

Then for a sufficiently large $m$, we have $F(x^{m},y^{m})<\inf\limits_{x\in
X}F(x,y^{m})+\eta$, i.e. $x^{m}\in G(y^{m})$ for a sufficiently large $m$.
Therefore, $G(y)$ is lower semi-continuous on $y$. ∎

Now we can prove the main theorem:

######  Theorem 14.

(Restate of Theorem 3) The unlearnable example game $\mathcal{G}$ has a
Stackelberg equilibrium.

###### Proof.

By Lemma 10, we have $J_{c}(\mathcal{A},\theta)$ is continuous, and by
Assumption 1, $\Theta$ and $\Gamma$ are compact. Then, for any $\mathcal{A}$,
$R_{\eta}(\mathcal{A})$ is nonempty. Let
$J(\theta)\coloneqq-\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{D}}\big{[}\mathcal{L}_{a}(x,y;\theta)\big{]}$.
Then the existence of a Stackelberg equilibrium is equivalent to
$\exists\mathcal{A}^{*}\in\Gamma$ such that:

$\sup_{\theta\in
R_{\eta}(\mathcal{A}^{*})}J(\theta)=\inf_{\mathcal{A}}\sup_{\theta\in
R_{\eta}(\mathcal{A})}J(\theta).$

By Lemma 13, we have that $R_{\eta}(\cdot)$ is lower semicontinuous. And since
$J(\theta)$ is lower semi-continuous on $\Theta$, by Lemma 12,
$V(\mathcal{A})=\sup\limits_{\theta\in R_{\eta}(\mathcal{A})}J(\theta)$ is
lower semicontinuous. Then $V(\mathcal{A})$ can reach the minimum as $\Gamma$
is compact. Then $\exists\mathcal{A}^{*}$ such that

$V(\mathcal{A}^{*})=\inf_{\mathcal{A}\in\Gamma}V(\mathcal{A})=\sup_{\theta\in
R_{\eta}(\mathcal{A}^{*})}J(\theta).$

Since $R_{\eta}(\mathcal{A})$ is a nonempty subset, there exists a
$\theta^{*}\in R_{\eta}(\mathcal{A}^{*})$. Thus, we have proved that there
exist $\mathcal{A}^{*}\in\Gamma$ and $\theta^{*}\in R_{\eta}(\mathcal{A}^{*})$
such that $(\mathcal{A}^{*},\theta^{*})$ is a Stacklberg equilibrium. ∎

### Proof of Lemma 7

######  Lemma 15.

(Restatement of Lemma 7) The adversarial loss function
$L_{adv}(x,y;f_{\theta})=\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}L_{ce}(f_{\theta}(x+\mu),y)$
is continuous.

###### Proof.

We show that $L_{adv}(x,y;f_{\theta})$ is continuous at $\theta$, and the
continuity for $x$ is similar. It is clear that $L_{ce}(f_{\theta}(x+\mu),y)$
is continuous on $\theta$. Therefore, for any $\gamma>0$, there exists a
$\Delta(x)>0$ (depending on $x$), such that for any
$||\theta_{1}-\theta_{2}||\leq\Delta(x)$ we have
$|L_{ce}(x,\theta_{1})-L_{ce}(x,\theta_{2})|\leq\gamma$.

Let
$L_{adv}(x_{1},\theta_{1})=\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}L_{ce}(f_{\theta_{1}}(x_{1}+\mu),y)$
and
$L_{adv}(x_{2},\theta_{2})=\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}L_{ce}(f_{\theta_{2}}(x_{2}+\mu),y)$.
Then for $||\theta_{1}-\theta_{2}||\leq\Delta(x_{2})$:

$\displaystyle L_{ce}(x_{1},\theta_{1})-L_{ce}(x_{2},\theta_{2})$
$\displaystyle\geq
L_{ce}(x_{2},\theta_{1})-L_{ce}(x_{2},\theta_{2})\geq-\gamma$

and for $||\theta_{1}-\theta_{2}||\leq\Delta(x_{1})$:

$\displaystyle L_{ce}(x_{1},\theta_{1})-L_{ce}(x_{2},\theta_{2})$
$\displaystyle\leq
L_{ce}(x_{1},\theta_{1})-L_{ce}(x_{1},\theta_{2})\leq\gamma.$

Thus, for any $\gamma>0$, there exists a constant
$\Delta=\min\\{\Delta(x_{1}),\Delta(x_{2})\\}>0$, such that for any
$||\theta_{1}-\theta_{2}||\leq\Delta$ we have
$|\max\limits_{x^{\prime}}L_{ce}(x^{\prime},\theta_{1})-\max\limits_{x^{\prime}}L_{ce}(x^{\prime},\theta_{2})|\leq\gamma$,
which means $L_{adv}(x,y;f_{\theta})$ is continuous at $\theta$. ∎

Similarly, we can prove that the trade-off adversarial loss is continuous:

######  Lemma 16.

The TRADES Loss function
$L_{tra}(x,y;f_{\theta})=L_{ce}(f_{\theta}(x),y)+\frac{1}{\lambda}\max\limits_{||\mu||_{\infty}\leq\epsilon_{d}}KL(f_{\theta}(x)||f_{\theta}(x+\mu))$
is continuous.

### Discrete Loss for Lowest Accuracy

As we stated in Remark 9, in order to exactly degrade the victim classifier’s
test accuracy, we consider a discrete loss function similar to (Gao, Liu, and
Yu 2022):

$\mathcal{L}_{acc}(f_{\theta}(x),y)=\begin{cases}0&\mathcal{L}_{cw}(f_{\theta}(x),y)\geq
0\\\ -1&\mathcal{L}_{cw}(f_{\theta}(x),y)<0\\\ \end{cases}$

where $\mathcal{L}_{cw}(f_{\theta}(x),y)=\max_{l\neq
y}[f_{\theta}(x)]_{l}-[f_{\theta}(x)]_{y}$ is the Carlini-Wagner loss (Carlini
and Wagner 2017). Then the negative value of population risk:
$-\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{D}}\big{[}\mathcal{L}_{acc}(x,y;\theta)\big{]}$
is exactly the classification accuracy.

Though here $\mathcal{L}_{acc}$ is not continuous, we also can extend our
existence Theorem 3 to this case:

######  Corollary 17.

Let $L_{a}=L_{acc}$, $L_{c}=L_{ce}$, we denote this Stackelberg game as
$\mathcal{G}_{acc}$. Then the unlearnable example game $\mathcal{G}_{acc}$ has
a Stackelberg equilibrium $(\mathcal{A}_{acc}^{*},\theta_{acc}^{*})$ .

###### Proof.

Since $L_{cw}(f_{\theta}(x),y)$ is continuous at $\theta$, it is easy to
verify that $-L_{acc}(f_{\theta}(x),y)$ is lower semi-continuous on $\theta$.
Furthermore $J(\theta)$ is lower semi-continuous on $\theta$. And the
remaining proof is similar to before. ∎

## Conclusion

While unlearnable example attacks have been well studied in various
formulations, a unified theoretical framework remains unexplored. In this
paper, we formulate the unlearnable example attack as a Stackelberg game that
encompasses a range of scenarios. We propose a novel attack GUE by directly
computing the Stackelberg equilibrium using a first-order optimization method
and by using a better loss function. Our approach possesses a stronger
theoretical intuition, whereas other methods are primarily based on empirical
practices. Furthermore, we show that the game equilibrium gives the most
powerful poison attack in the sense that the victim neural network has the
lowest test accuracy among all networks within the same hypothesis space.

Extensive experiments demonstrate the effectiveness of GUE in different
scenarios, in particular, the generalizability of poison generator trained
with low percentage of training data and its effectiveness against adversarial
training.

#### Limitations

Our algorithm cannot converge well when the adversarial radius is larger,
hence not effective to poison adversarial trained model with more aggressive
budgets. Future research is needed to find more effective algorithms to solve
the unlearnable example games.

## Acknowledgements

This work is partially supported by the NSFC grant No.11688101 and NKRDP grant
No.2018YFA0306702. The authors thank anonymous referees for their valuable
comments.

## References

  * Aubin and Ekeland (1984) Aubin, J.-P.; and Ekeland, I. 1984.  _Applied Nonlinear Analysis_.  Wiley, New York.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Carlini and Wagner (2017) Carlini, N.; and Wagner, D. 2017.  Towards evaluating the robustness of neural networks.  In _2017 ieee symposium on security and privacy (sp)_ , 39–57. Ieee.
  * DeVries and Taylor (2017) DeVries, T.; and Taylor, G. W. 2017.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_.
  * Feng, Cai, and Zhou (2019) Feng, J.; Cai, Q.-Z.; and Zhou, Z.-H. 2019.  Learning to confuse: generating training time adversarial data with auto-encoder.  _Advances in Neural Information Processing Systems_ , 32.
  * Fowl et al. (2021) Fowl, L.; Goldblum, M.; Chiang, P.-y.; Geiping, J.; Czaja, W.; and Goldstein, T. 2021.  Adversarial Examples Make Strong Poisons.  In _Advances in Neural Information Processing Systems_ , volume 34, 30339–30351.
  * Fu et al. (2022) Fu, S.; He, F.; Liu, Y.; Shen, L.; and Tao, D. 2022.  Robust unlearnable examples: Protecting data against adversarial learning.  _arXiv preprint arXiv:2203.14533_.
  * Gao, Liu, and Yu (2022) Gao, X.-S.; Liu, S.; and Yu, L. 2022.  Achieving optimal adversarial accuracy for adversarial deep learning using Stackelberg games.  _Acta Math Sci_ , 2399–2418.
  * Gong, Liu, and Liu (2021) Gong, C.; Liu, X.; and Liu, Q. 2021.  Automatic and harmless regularization with constrained and lexicographic optimization: A dynamic barrier approach.  _Advances in Neural Information Processing Systems_ , 34: 29630–29642.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Huang et al. (2021) Huang, H.; Ma, X.; Erfani, S. M.; Bailey, J.; and Wang, Y. 2021.  Unlearnable examples: Making personal data unexploitable.  _arXiv preprint arXiv:2101.04898_.
  * Jacot, Gabriel, and Hongler (2018) Jacot, A.; Gabriel, F.; and Hongler, C. 2018.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Liu et al. (2022) Liu, B.; Ye, M.; Wright, S.; Stone, P.; and Liu, Q. 2022.  Bome! bilevel optimization made easy: A simple first-order approach.  _Advances in Neural Information Processing Systems_ , 35: 17248–17262.
  * Lu, Kamath, and Yu (2022) Lu, Y.; Kamath, G.; and Yu, Y. 2022.  Indiscriminate Data Poisoning Attacks on Neural Networks.  _arXiv preprint arXiv:2204.09092_.
  * Madry et al. (2017) Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Proceedings, Part III 18_ , 234–241. Springer.
  * Sandoval-Segura et al. (2022a) Sandoval-Segura, P.; Singla, V.; Fowl, L.; Geiping, J.; Goldblum, M.; Jacobs, D.; and Goldstein, T. 2022a.  Poisons that are learned faster are more effective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 198–205.
  * Sandoval-Segura et al. (2022b) Sandoval-Segura, P.; Singla, V.; Geiping, J.; Goldblum, M.; Goldstein, T.; and Jacobs, D. 2022b.  Autoregressive perturbations for data poisoning.  _Advances in Neural Information Processing Systems_ , 35: 27374–27386.
  * Tao et al. (2021) Tao, L.; Feng, L.; Yi, J.; Huang, S.-J.; and Chen, S. 2021.  Better safe than sorry: Preventing delusive adversaries with adversarial training.  _Advances in Neural Information Processing Systems_ , 34: 16209–16225.
  * van Vlijmen et al. (2022) van Vlijmen, D.; Kolmus, A.; Liu, Z.; Zhao, Z.; and Larson, M. 2022.  Generative Poisoning Using Random Discriminators.  _arXiv preprint arXiv:2211.01086_.
  * Wen et al. (2023) Wen, R.; Zhao, Z.; Liu, Z.; Backes, M.; Wang, T.; and Zhang, Y. 2023.  Is Adversarial Training Really a Silver Bullet for Mitigating Data Poisoning?  In _The Eleventh International Conference on Learning Representations_.
  * Yu et al. (2022) Yu, D.; Zhang, H.; Chen, W.; Yin, J.; and Liu, T.-Y. 2022.  Availability attacks create shortcuts.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2367–2376.
  * Yuan and Wu (2021) Yuan, C.-H.; and Wu, S.-H. 2021.  Neural tangent generalization attacks.  In _International Conference on Machine Learning_ , 12230–12240. PMLR.
  * Yun et al. (2019) Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y. 2019.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 6023–6032.
  * Zhang et al. (2017) Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_.
  * Zhang et al. (2019) Zhang, H.; Yu, Y.; Jiao, J.; Xing, E.; El Ghaoui, L.; and Jordan, M. 2019.  Theoretically principled trade-off between robustness and accuracy.  In _International conference on machine learning_ , 7472–7482. PMLR.


# Building a Corpus for Biomedical Relation Extraction of Species Mentions

Oumaima El Khettari, Solen Quiniou, Samuel Chaffron
Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000
{oumaima.el-khettari, solen.quiniou<EMAIL_ADDRESS>

###### Abstract

We present a manually annotated corpus, Species-Species Interaction, for
extracting meaningful binary relations between species, in biomedical texts,
at sentence level, with a focus on the gut microbiota. The corpus leverages
PubTator to annotate species in full-text articles after evaluating different
Named Entity Recognition species taggers. Our first results are promising for
extracting relations between species using BERT and its biomedical variants.

##  1 Introduction

The field of biomedical relation extraction (RE) has made significant
advancements in recent years, with the development of various state-of-the-art
models for extracting meaningful relationships between entities from
scientific articles. However, the availability of annotated datasets for
specific types of relations, such as interactions between species, remains
limited. Studying the interactions and relationships between different species
is of great interest in many biomedical research areas, including gut
microbiome research. The gut microbiota, a complex community of microorganisms
residing in the human gastrointestinal tract, plays a crucial role in human
health and disease. Understanding the interactions and relationships between
different species within the gut microbiome is of significant importance in
advancing knowledge on microbiome-mediated health outcomes.

The first challenge in studying this type of interactions is the lack of
annotated datasets that specifically capture the semantic expressions of
interactions between species. Most studied interactions occur between
proteins, drugs, genes, and diseases. The representations of these relations
may not fully capture the complexity of species interactions, particularly in
the specific context of the microbiome. In this article, we present a
dedicated corpus for relation extraction of species mentions in biomedical
texts, by manually annotating binary relations at the sentence level in an
existing corpus for Named Entity Recognition (NER) and in a new corpus
focusing on the gut microbiome. We also choose to leverage PubTator, a
reliable system for annotating species in full-text articles, after evaluating
its performance with respect to a fine-tuned BERT and a scispaCy model. We
carefully selected a set of PMC documents from PubTator based on heuristics
that align with our domain of interest and annotated a selection of sentences
from the corpus with binary relations, consisting of the presence/absence of a
relation. Afterwards, we proceeded to fine-tune existing transformer-based
models on our corpus to highlight the impact of a new small set of semantic
relation expressions. Our contributions are as follows:

  * •

A study of the Species entities in the literature;

  * •

Species-Species Interaction (SSI), a corpus of manually annotated binary
relations on full-text scientific articles, publicly available on
HuggingFace111https://huggingface.co/datasets/taln-ls2n/SSI;

  * •

A comparative study of the SOTA Species NER taggers;

  * •

Initial results on extracting relations between species using BERT and several
of its biomedical variants.

##  2 Related Work

###  2.1 NER for Species

The presence of relations in a text directly implies the presence of named
entities of interest. Our work is based on the hypothesis stating the
following: a relation cannot be expressed in a text unless at least two named
entities are mentioned in the same text. Prior work involving the
investigation of species mentions has been conducted in order to evaluate
existing methods of NER, for the entity type "Species".

Little research has been conducted on the specific Species mention in the
context of NER and RE. This scarcity can be attributed primarily to the lack
of resources specifically designed for Species. First, methods applied for the
mention Species were rule-based systems along with dictionary-based models,
together with statistical methods, such as TaxonFinder Wei et al. (2013) and
MetaMap Aronson (2006). This type of method strongly depends on the
completeness of ontologies, knowledge graphs, and dictionaries. Then, deep
learning methods requiring massive amounts of annotated data, relied
exclusively on a few available corpora of species mentions to fine-tune
several biomedical variants of BERT Devlin et al. (2019), such as BioBERT Lee
et al. (2019).

In this context, scispaCy Neumann et al. (2019), a library specifically
designed for biomedical and scientific text mining, offers a variety of NER
models, built on the deep learning spaCy models. PubTator Central Wei et al.
(2019), a web-based tool designed to automatically annotate and extract
information from biomedical literature, provides access to more than 3 million
full-text articles that have been annotated with different entities, including
Species Comeau et al. (2019). Its annotations are generated using a
combination of dictionary- and machine learning-based models trained on large
biomedical data.

In terms of corpora for NER, LINNAEUS Gerner et al. (2010) and S800 Pafilis et
al. (2013) are the most widely known and used corpora for Species mentions as
they served for the fine-tuning of deep learning models. LINNAEUS consists of
100 randomly selected and manually annotated full-text documents from the PMC
OA document set, resulting in 2 988 annotations. As for S800, it is a
collection of 800 randomly collected and manually annotated abstracts from
journals, classified into 8 categories, and including 3 708 annotations.

###  2.2 Relation Extraction in Biomedical Literature

Relation Extraction is a task where the goal is to identify and classify the
relationships between pairs of entities. This specific task is modeled as a
classification problem. In binary relation extraction, the task is to
determine whether a relationship exists between two entities in a given
sentence. The goal is to classify the sentence as either containing a
relationship or not. Another task is to identify the type of relationship
between two entities, among some predefined categories. For each type of
relation extraction, a suitable corpus has to be defined.

There are several existing biomedical RE corpora that have been widely used
for the training and fine-tuning of models, where a variety of interaction
types are captured. For most of these corpora, the provided annotations are
binary ones, indicating the presence or absence of a relation, at the sentence
level, for protein-protein interactions (PPI) Hamilton et al. (2017), drug-
drug interactions (DDI) Segura-Bedmar et al. (2013), gene-disease interactions
(GAD) Bravo et al. (2015), as well as EU-ADR Van Mulligen et al. (2012).

Despite the large diversity of the above entities in existing datasets, the
mention Species remains off the list. Consequently, interactions between
species remain under-represented, and the diverse nature of species-related
information, specifically in scientific articles, including taxonomy
differences, evolutionary relationships, and ecological interactions, further
complicates the task of relation extraction in this domain.

Biomedical RE dataset labeling is known to be challenging, often time-
consuming, and requiring domain expertise. As a result, manually labeled
datasets are often size limited. Due to these challenges, distant supervision
has been introduced as a strategy for data augmentation in biomedical RE Mintz
et al. (2009). While this strategy proved its efficiency for some use cases,
it hinders the extraction of semantic relations. As for NER, a variety of
methods can perform RE, starting from rule-based models to transformer-based
ones. BERT and its biomedical variations are achieving promising results for
this task, especially on the binary relation extraction task, using the
corpora previously cited.

##  3 Relation Extraction Corpus for Species

The participating entities in known interactions are major biomedical
concepts, such as genes, proteins, diseases, and drugs. As we focus on
studying relations occurring between species in scientific research articles,
we encountered significant challenges. One major challenge is the scarcity of
annotated corpora that capture the nuanced semantic expressions of
interactions between species, which are distinct from the well-studied RE
datasets. As a first step to tackle this problem, we introduced Species-
Species Interaction (SSI), a manually annotated corpus dedicated to species
binary interactions, built from full-text scientific articles.

###  3.1 Selecting Initial Corpora

Our main goal is to extract information from full-text articles, as we aim to
detect interactions that are poorly represented and may not be found in
smaller parts of texts, like abstracts. We were interested, at the same time,
by extending existing corpora mentioning Species (aka S800 and LINNAEUS) by
adding Species-Species relations as well as creating a new corpus specifically
dedicated to the gut microbiome, using PubTator to provide Species mentions.

The selection of PubTator articles was thus based on the following query:
<"gut microbiota" OR "gut microbiome" OR "intestinal microbiota" OR
"intestinal microbiome">. Then, to select relevant articles for our domain, we
established heuristics taking into account the number of species mentions in
the NCBI taxonomy Schoch et al. (2020) and the diversity of these annotations.
We also considered the hierarchical structure of the taxonomy, as common non-
specialized vocabulary terms like "human," "patient," or "mice" are frequently
used in scientific articles. As these terms most often refer to model
organisms, they may not be relevant to our study.

###  3.2 Annotating Species-Species Relation

The step of collecting data is followed by splitting the articles into
sentences using spaCy, since the annotation is at the sentence level. We
focused on sentences where exactly two species are mentioned to maximize the
probability that a binary relation is indeed expressed. Consequently, no
sentences with the previous criterion were found on S800. This is due to the
sparsity of entities mentions in abstracts. Therefore, we collected 442
sentences from the LINNAEUS corpus and 557 sentences from the corpus obtained
with PubTator, to obtain our SSI corpus.

#### Annotation Guidelines

We aspired to capture the semantic representation of interactions between
species, in sentences. We refer to interactions between two species as various
mechanisms through which the species impact one another’s behaviour, survival,
or the overall ecological dynamics within an ecosystem.

Before the annotation process, we applied the mask @SPECIES$ on the species
mentions in all sentences. The objective here was to limit the impact of
external knowledge, like the name of a species, on the decision of the
presence of a relation. We finally assigned the label 1 to sentences
containing a relation between at least two species mentions (to denote the
presence of a relation) and the label 0 when no relation seems to appear in
the sentences.

Sentences |  Label
---|---
_Effects of dark @SPECIES$ and @SPECIES$ consumption on endothelial function and arterial stiffness in overweight adults_ |  0
_One @SPECIES$ (Epi132) revealed family cancer occurrence resembling families harboring CHEK2 mutations in general, the other @SPECIES$ (epi203) was non-conclusive._ |  0
_Eosinophil accumulation induced by @SPECIES$ interleukin-8 in the @SPECIES$ in vivo._ |  1
_@SPECIES$ ellagitannins thwarted the positive effects of dietary fructo-oligosaccharides in @SPECIES$ cecum._ |  1
Table 1: SSI annotation examples

In table 1, we give examples of annotated sentences. A set of annotation
guidelines have been defined so as to achieve the previously mentioned goal
while annotating, including the following:

  * •

Contextual indicators: The first step of the annotation process is analyzing
the sentence for any contextual cues suggesting a relation between the
species. This can include explicit keywords, verbs, or phrases that imply a
connection, such as "interacts with", "physically related to", "association".

  * •

Negation and absence: Negation is considered as a type of relation. Thereby,
sentences with a clear identifiable negation should be annotated with label 1.

  * •

Comparison: Sentences claiming a comparison between the mentioned species
should be labeled as 1 only if they share a clearly cited characteristic.

  * •

Multiple relations: Only consider the potential presence or absence of a
relation between the masked species.

Table 2 gives the statistics on the train, validation, and test splits of our
SSI corpus. We used 60% of the corpus for the train set and we equally split
the remaining 40% on the test and validation sets.

SSI | #train | #val | #test
---|---|---|---
label 1 | 300 | 93 | 109
label 0 | 299 | 107 | 91
Table 2: Statistics on our corpus where label 0 states for the absence of a
relation and label 1 for its presence

##  4 Experiments

We first conducted experiments on NER taggers for the Species mentions and
then evaluated BERT-like models for the RE tasks on the SSI corpus.

###  4.1 Evaluating NER Taggers on Species

As a preliminary study of the Species RE and in order to add NER annotations
to our RE corpus, we compared the performance of fine-tuned BERT on LINNAEUS,
scispaCy model, and PubtaTor on S800 for the task of NER. One of NER scispaCy
models is en_ner_craft_md. It is trained on the CRAFT corpus Bada et al.
(2012) and it identifies various biomedical entities, including taxons, a
close term to species. A taxon refers to a group of organisms classified based
on shared characteristics, while species is the basic unit of biological
classification consisting of individuals that can interbreed and produce
fertile offspring. Due to the proximity of these two concepts, we consider
taxon a reference to species.

In addition to precision, recall, and F1-score, we report the Slot Error Rate
(SER). It is a metric that complements the previous measurements by
considering various factors, including deletions, insertions, and boundary-
related errors.

During the comparison of the collected named entities, an issue was
encountered while matching S800 articles with their versions from PubTator due
to encoding problems. As a result, the text was garbled or displayed as
gibberish characters, with certain letter combinations replaced with symbols.
This was particularly problematic when the affected text was a named entity.
To address this issue, spaces were corrected and the texts were categorized
into two classes: the first containing the majority of articles with matching
versions, and the second containing articles with encoding problems. The
evaluation was performed on the first category, containing 2924 entity.

Models | P | R | F1 | SER
---|---|---|---|---
Fine-Tuned BERT | 0.53 | 0.62 | 0.57 | 0.75
ScispaCy | 0.22 | 0.22 | 0.22 | 1.35
PubTator | 0.68 | 0.74 | 0.71 | 0.51
Table 3: Comparing NER models on Species mentions

As shown in Table 3, PubTator outperforms fine-tuned BERT on Linnaeus as well
as the scispaCy model. It should be mentioned that the metrics are calculated
on the exact matches of the named entities. However, the SER value, reaching a
minimal value of 51%, sheds light on the complexity of handling composed or
long species mentions. Nevertheless, PubtaTor handles better these instances
compared to the remaining systems. For this reason, we decided to use PubtaTor
for the annotation of Species in our corpus.

###  4.2 Comparing models for RE on Species

We performed RE by fine-tuning existing models on the SSI corpus. Along with
BERT-base, we used the following models, chosen for their promising results on
other biomedical RE corpora:

  * •

BioBERT, pre-trained on abstracts from the PubMed database and full-text
articles from the PubMed Central Open Access Subset;

  * •

SciBERT Beltagy et al. (2019), pre-trained on Semantic Scholar full-text
articles, with a majority belonging to the biomedical domain;

  * •

BioLinkBERT Yasunaga et al. (2022), pre-trained on PubMed abstracts as well as
their citations in order to enrich the model with the existing dependencies
between the academic articles.

We report, in Table 4, the results on the test set of SSI, before and after
the fine-tuning of the compared models.

Prior to the fine-tuning on SSI, one major observation was that all models
performed poorly on label 0, except BioLinkBERT. This is due to the fact that
there are larger possibilities of semantic expressions for no relation. This
is not the case for the second category as the set of relations between
species is finite. On the contrary, BiolinkBERT achieved a very low recall on
label 1, a tendency that was altered after the fine-tuning by gaining points
on the F1 score. Overall, the results indicated that fine-tuning pre-trained
models led to a notable improvement in performance, as the precision and the
F1 scores were higher for all the fined-tuned models as compared to their pre-
trained counterparts.

Models | Label 0 | Label 1
---|---|---
| P | F1 | P | F1
BERT base | 0.33 | 0.02 | 0.54 | 0.69
BioBERT | 0.39 | 0.26 | 0.55 | 0.62
SciBERT | 0.37 | 0.34 | 0.49 | 0.52
BioLinkBERT | 0.45 | 0.62 | 0.50 | 0.02
FT BERT base | 0.72 | 0.71 | 0.76 | 0.76
FT BioBERT | 0.64 | 0.68 | 0.75 | 0.70
FT SciBERT | 0.69 | 0.71 | 0.76 | 0.75
FT BioLinkBERT | 0.66 | 0.70 | 0.77 | 0.72
Table 4: Results of RE before and after fine-tuning models on SSI (fine-tuned
models are denoted FT)

###  4.3 Error Analysis

Given the multiple demonstrations of the effectiveness of fine-tuning in
enhancing downstream tasks, we conducted an error analysis on the test sample
for each model. By this mean, we aimed to gain insights into the performance
of the models, as well as the processing of the new corpus.

Upon analyzing the errors made by all the models, it became apparent that the
majority of confusion arised when dealing with enumerated species mentions. In
these cases, our annotation required the presence of a relation only if the
enumerated species share a distinctly mentioned characteristic. However,
accurately identifying and highlighting such characteristics poses a
challenge. Errors also occurred in instances where the terms "relationship,"
"interaction," or negation indicators are explicitly stated in the text.
Despite their explicit mention, the semantic information may not indicate a
genuine interaction between the two species or clearly expresses a negation.
Another significant concern pertains to co-references when examining examples
with co-referent species mentions. According to the annotation, these
instances are marked as having a relation. However, models tend to classify
them as 0 due to the semantic challenge of identifying indicators that signify
a relation. Furthermore, our findings indicate that there is an increased
likelihood of mislabeling, particularly in long sentences, as the distance
between two species mentions grows in terms of the number of words.
Additionally, there is an error type closely associated with named entities.
When the masked species mention is part of a compound name, it poses a
challenge for the models to determine whether the relation is solely between
the masked entity or the entire named entity.

Addressing these challenges is essential to enhance the performance of the
models. Major indicators of improvement in our case include focusing on co-
reference resolution, expanding the dataset by incorporating enumeration
instances, varying the length of sentences, and further enhancing named entity
recognition systems for the mention Species.

##  5 Conclusion

We introduced a new corpus called Species-Species Interaction (SSI) for
extracting meaningful binary relations between species in biomedical texts,
with a focus on the gut microbiome, to address the absence of Species-related
corpora. We included a comparative study of species NER taggers followed by
initial results on extracting relations using BERT and its biomedical
variants. Our experiments demonstrated that performances of pre-trained models
were improved with fine-tuning, despite the use of a limited-size corpus. To
gain a better understanding of these results, we performed an error analysis,
allowing us to identify specific areas for improvement that should be taken
into consideration. In the future, we will aim to enlarge our SSI corpus by
further annotating sentences and also by integrating relation types to better
represent Species-Species interactions.

##  6 Limitations

The methodology of the constructed corpus is based on NER, making SSI a corpus
integrating RE with NER. We aimed to enrich our experiments with multi-task
learning so as to measure the impact of NER on the extraction of relations.
Unfortunately, after conducting statistical analysis on SSI, it showed that,
due to its limited size, we did not have enough instances of the same pair of
entities participating in one sentence. As this statement would negatively
bias our multi-task model, we refrained from conducting these experiments,
before proceeding to annotate further instances and add them to our corpus in
further work. In terms of annotation, only one annotator has been invested in
the current task, as it is a preliminary work including a new dataset for
relation extraction, with new entities and relation types. In a future work,
multiple annotators performing the annotation task and metrics for the corpus
quality will be presented.

## Acknowledgements

We would like to express our gratitude to Gesser Riahi and Adrien Lesenechal
for their valuable contributions to this work. Their commitment was
instrumental in the successful completion of this project. We would also like
to thank the anonymous reviewers for their valuable inputs to this article.

This work was financially supported by the ANR AIBy4 project
(ANR-20-THIA-0011) and Nantes Université.

## References

  * Aronson (2006) Alan R Aronson. 2006.  MetaMap: Mapping Text to the UMLS Metathesaurus.  _Bethesda, MD: NLM, NIH, DHHS_ , 1:1–26.
  * Bada et al. (2012) Michael Bada, Miriam Eckert, Donald Evans, Kristin Garcia, Krista Shipley, Dmitry Sitnikov, William A Baumgartner, K Bretonnel Cohen, Karin Verspoor, Judith A Blake, et al. 2012.  Concept Annotation in the CRAFT Corpus.  _BMC bioinformatics_ , 13(1):1–20.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China.
  * Bravo et al. (2015) Àlex Bravo, Janet Piñero, Núria Queralt-Rosinach, Michael Rautschka, and Laura I Furlong. 2015.  Extraction of Relations Between Genes and Diseases from Text and Large-Scale Data Analysis: Implications for Translational Research.  _BMC Bioinformatics_ , 16(1).
  * Comeau et al. (2019) Donald C Comeau, Chih-Hsuan Wei, Rezarta Islamaj Doğan, and Zhiyong Lu. 2019.  PMC Text Mining Subset in BioC: About Three Million Full-Text Articles and Growing.  _BMC Bioinformatics_ , 35(18):3533–3535.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 4171–4186, Minneapolis, MN, USA.
  * Gerner et al. (2010) Martin Gerner, Goran Nenadic, and Casey M Bergman. 2010.  LINNAEUS: a Species Name Identification System for Biomedical Literature.  _BMC Bioinformatics_ , 11(1):1–17.
  * Hamilton et al. (2017) William L. Hamilton, Rex Ying, and Jure Leskovec. 2017.  Inductive Representation Learning on Large Graphs.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17)_ , page 1025–1035, Red Hook, NY, USA.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a Pre-Trained Biomedical Language Representation Model for Biomedical Text Mining.  _BMC Bioinformatics_ , 36(4):1234–1240.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009.  Distant Supervision for Relation Extraction Without Labeled Data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011.
  * Neumann et al. (2019) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019.  ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 319–327, Florence, Italy.
  * Pafilis et al. (2013) Evangelos Pafilis, Sune P Frankild, Lucia Fanini, Sarah Faulwetter, Christina Pavloudi, Aikaterini Vasileiadou, Christos Arvanitidis, and Lars Juhl Jensen. 2013\.  The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text.  _PloS one_ , 8(6):e65390.
  * Schoch et al. (2020) Conrad L Schoch, Stacy Ciufo, Mikhail Domrachev, Carol L Hotton, Sivakumar Kannan, Rogneda Khovanskaya, Detlef Leipe, Richard Mcveigh, Kathleen O’Neill, Barbara Robbertse, et al. 2020.  NCBI Taxonomy: a Comprehensive Update on Curation, Resources and Tools.  _Database_ , 2020.
  * Segura-Bedmar et al. (2013) Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. 2013.  SemEval-2013 Task 9 : Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013).  In _Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 341–350, Atlanta, GE, USA.
  * Van Mulligen et al. (2012) Erik M Van Mulligen, Annie Fourrier-Reglat, David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A Kors, and Laura I Furlong. 2012.  The EU-ADR Corpus: Annotated Drugs, Diseases, Targets, and their Relationships.  _Journal of biomedical informatics_ , 45(5):879–884.
  * Wei et al. (2019) Chih-Hsuan Wei, Alexis Allot, Robert Leaman, and Zhiyong Lu. 2019.  PubTator central: Automated Concept Annotation for Biomedical Full Text Articles.  _Nucleic acids research_ , 47(W1):W587–W593.
  * Wei et al. (2013) Qin Wei, Chris Freeland, and P Bryan Heidorn. 2013.  Name Matters: Taxonomic Name Recognition (TNR) in Biodiversity Hheritage Library (BHL).  _iConference 2010 Proceedings. University of Illinois_ , 2010.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  LinkBERT: Pretraining Language Models with Document Links.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 8003–8016, Dublin, Ireland.

##  Appendix A Training settings

  * •

GPU type: TITAN X 12Go

  * •

Text max size: 512

  * •

Learning rate: $1\text{\times}{10}^{-5}$

  * •

Batch size: 8

  * •

Number of epochs: 10

  * •

Optimizer: AdamW