# Open World Classification with Adaptive Negative Samples

Ke Bai1, Guoyin Wang2, Jiwei Li3, Sunghyun Park2,
Sungjin Lee2, Puyang Xu2, Ricardo Henao1,4, Lawrence Carin4
1Duke University 2Amazon 3Zhejiang University 4KAUST
{ke.bai<EMAIL_ADDRESS>{guoyiwan, sunghyu, sungjinl,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Open world classification is a task in natural language processing with key
practical relevance and impact. Since the open or unknown category data only
manifests in the inference phase, finding a model with a suitable decision
boundary accommodating for the identification of known classes and
discrimination of the open category is challenging. The performance of
existing models is limited by the lack of effective open category data during
the training stage or the lack of a good mechanism to learn appropriate
decision boundaries. We propose an approach based on adaptive negative samples
(ANS) designed to generate effective synthetic open category samples in the
training stage and without requiring any prior knowledge or external datasets.
Empirically, we find a significant advantage in using auxiliary one-versus-
rest binary classifiers, which effectively utilize the generated negative
samples and avoid the complex threshold-seeking stage in previous works.
Extensive experiments on three benchmark datasets show that ANS achieves
significant improvements over state-of-the-art methods.

##  1 Introduction

Standard supervised classification assumes that all categories expected in the
testing phase have been fully observed while training, i.e., every sample is
assigned to one known category as illustrated in Figure 1(a). This may not
always be satisfied in practical applications, such as dialogue intention
classification, where new intents are expected to emerge. Consequently, it is
desirable to have a classifier capable of discriminating whether a given
sample belongs to a known or an unknown category, e.g., the red samples in
Figure 1(a). This problem can be understood as a $(C+1)$ classification
problem, where $C$ is the number of known categories and the additional
category is reserved for open (unknown) samples. This scenario is also known
as multi-class open set recognition Scheirer et al. (2014).

Figure 1: Illustration of previous methods and our proposed one. $C0$, $C1$,
and $C2$ are three categories. The boundary is used to discriminate known and
open (unknown) categories. (a) Boundary learned by supervised learning. (b)
Optimal decision boundary. (c) ADB method which has a closed decision
boundary, but may capture irrelevant points. (d) Proposed ANS method.

To discriminate the known from the open samples during inference, it is
necessary to create a clear classification boundary that separates the known
from the open category. However, the lack of open category samples during
training makes this problem challenging. Current research in this setting
mainly focuses on two directions. The first direction mainly estimates a
tighter decision boundary between known classes to allow for the possibility
of the open category. Existing works in this direction include the Local
Outlier Factor (LOF) (Breunig et al., 2000; Zhou et al., 2022; Zeng et al.,
2021), Deep Open Classification (DOC) (Shu et al., 2017) and Adaptive Decision
Boundary (ADB) (Zhang et al., 2021b). LOF and ADB calibrate the decision
boundary in the feature space while DOC does it in the probability space.

The second direction deals with learning a better feature representation to
make the boundary-seeking problem easier. In this direction, DeepUnk (Lin and
Xu, 2019a) and SEG (Yan et al., 2020) added constraints to the feature space,
SCL (Zeng et al., 2021) and Zhou et al. (2022) fine-tuned the feature
extractor backbone with contrastive learning. Zhan et al. (2021) considered
introducing open samples from other datasets as negative, and Shu et al.
(2021) generated samples with contradictory meanings using a large pretrained
model. The latter two deliver large performance gains.

These improvements demonstrate the significance of negative samples in
determining the boundary between the known and open categories. To accomplish
the same in the absence of additional datasets or knowledge, we propose a
novel negative-sample constraint and employ a gradient-based method to
generate pseudo open category samples. As shown in Figure 1(d), negative
samples are adaptively generated for each category to closely bound each
category.

Given the generated negative samples, we then empirically find that using
auxiliary one-versus-rest binary classifiers can better capture the boundary
between the known and the open category, relative to a $(C+1)$-way classifier
(Zhan et al., 2021), where all the open categories, possibly distributed in
multiple modes or arbitrarily scattered over the feature space, are
categorized into one class.

Specifically, we first learn a $C$-category classifier on known category data.
Then for each known category, we learn an auxiliary binary classifier,
treating this category as positive and others as negative. During inference,
one sample is recognized as open if all the binary classifiers predict it as
negative, thus not belonging to any known category

Our main contributions are summarized below:

  * •

We propose a novel adaptive negative-sample-generation method for open-world
classification problems without the need for external data or prior knowledge
of the open categories. Moreover, negative samples can be added to existing
methods and yield performance gains.

  * •

We show that synthesized negative samples combined with auxiliary one-versus-
rest binary classifiers facilitate learning better decision boundaries and
requires no tuning (calibration) on the open category threshold.

  * •

We conduct extensive experiments to show that our approach significantly
improves over previous state-of-the-art methods.

##  2 Related Work

Boundary Calibration The classical local outlier factor (LOF) (Breunig et al.,
2000) method is a custom metric that calculates the local density deviation of
a data point from its neighbors. However, there is not a principled rule on
how to choose the outlier detection threshold when using LOF. Zeng et al.
(2021); Zhou et al. (2022) added open category data into the validation set to
estimate or grid-search the proper threshold. So motivated, Bendale and Boult
(2016) fit the output logits of the classifier to a Weibull distribution, but
still use a validation set that contains the open category to select the
confidence threshold. Further, Shu et al. (2017) employed one-versus-rest
binary classifiers and then calculates the threshold over the confidence score
space by fitting it to a Gaussian distribution. This method is limited by the
often inaccurate (uncalibrated) predictive confidence learned by the neural
network (Guo et al., 2017). Adaptive decision boundary (Zhang et al., 2021b),
illustrated in Figure 1(c), was recently proposed to learn bounded spherical
regions for known categories to contain known class samples. Though this post-
processing approach achieves state-of-the-art performance, it still suffers
from the issue that the tight-bounded spheres may not exist or cannot be well-
defined in representation space. Due to the fact that high-dimensional data
representations usually lie on a low-dimensional manifold (Pless and Souvenir,
2009), a sphere defined in a Euclidean space can be restrictive as a decision
boundary. Moreover, the issue can be more severe if certain categories follow
multimodal or skewed distributions.

Figure 2: Illustration of the proposed ANS algorithm. (a) The training stage
is divided into two blocks. Top: known class classification described in Sec.
3.1. Bottom: one-versus-rest classification (Sec. 3.2) on Category 1 as an
example. The negative samples are from two sources, namely, known from other
classes (in red) and synthesized samples (in yellow). (b) Inference arm
described at the end of Section 3.2.

Representation Learning DeepUnk (Lin and Xu, 2019a) trains the feature
extractor with Large Margin Cosine Loss (Wang et al., 2018). SEG (Yan et al.,
2020) assumes that the known features follow the mixture Gaussian
distribution. Zeng et al. (2021) and Zhou et al. (2022) applied supervised
contrastive learning (Chen et al., 2020) and further improve the
representation quality by using $k$-nearest positive samples and negative
samples collected from the memory buffer of MOCO (He et al., 2020). Getting a
better representation trained with known category data only is complementary
to our work, since a better pretrained backbone can further improve our
results. Recent works found that it may be problematic to learn features
solely based on the known classes; thus, it is crucial to provide samples from
unknown classes during training. Specifically, Zhan et al. (2021) creates
negative samples with mixup of training data and examples from an external
dataset. Shu et al. (2021) generates open class samples using BART (Lewis et
al., 2019) and external text entailment information.

##  3 Methodology

#### Problem Definition

Suppose we are given a training dataset
$\mathcal{D}=\\{({\bm{x}}_{1},y_{1}),({\bm{x}}_{2},y_{2}),\ldots,$
$({\bm{x}}_{N},y_{N})\\}$ consisting of $N$ examples, where ${\bm{x}}_{i}$ is
an input text sequence and $y_{i}$ its corresponding label, which belongs to a
predefined set $\mathcal{L}=\\{l_{1},l_{2},\ldots,l_{C}\\}$ with $C$
categories, thus $y_{i}\in\mathcal{L},\forall i\in[N]$, where
$[N]\coloneqq[1,\ldots,N]$. In this paper, we use $[\cdot]$ to represent index
sequence. In an open world classification setting, the goal is to learn a
model which categorizes a test instance to either one of the predefined $C$
categories or as an open category. In practice, the open category is denoted
as a unique category $l_{0}$.

###  3.1 Known Category Classification

Following the setting suggested in Lin and Xu (2019b); Zhang et al. (2021b),
we use BERT (Devlin et al., 2018) as our feature extractor
$f_{\psi_{enc}}(\cdot)$. For each input text sequence ${\bm{x}}$, where
${\bm{x}}$ is represented as a sequence of tokens
$[t_{1},t_{2},\ldots,t_{n}]$, we take the average of features $z_{t_{i}}$ of
each token $t_{i}$ extracted from the BERT output layer as the sentence
representation ${\bm{z}}$. - The training of $f_{\psi_{enc}}(\cdot)$ is
formulated as a multi-category classification problem by minimizing the cross-
entropy loss $\mathcal{L}^{cls}$:

$\displaystyle\mathcal{L}^{cls}(\psi_{enc},\psi_{cls})=$ (1)
$\displaystyle\hskip
56.9055pt-\sum_{i\in[N]}\log\frac{\exp(f_{\psi_{cls}}^{y_{i}}({\bm{z}}_{i}))}{\sum_{j=1}^{C}\exp(f_{\psi_{cls}}^{j}({\bm{z}}_{i}))},$

where $f_{\psi_{cls}}(\cdot)$ is a classifier that takes ${\bm{z}}$ as input
and the output dimension is the number of known categories $C$.
$f_{\psi_{cls}}^{j}({\bm{z}})$ represents the output logit for the $j$-th
category. A well-trained feature extractor $f_{\psi_{enc}}(\cdot)$ and a high-
quality classifier $f_{\psi_{cls}}(\cdot)$ can extract representative features
of each category and ensure good performance on the classification of known
categories during the inference stage.

###  3.2 Open Category Recognition

Once the classifier for the known categories is available, the task is to
recognize samples from the open category versus the ones from known
categories. As mentioned above, directly using the known category classifier
$f_{\psi_{cls}}(\cdot)$ can result in poor performance (Hendrycks and Gimpel,
2016), while using a $(C+1)$ category classifier setting is complicated due to
the need to find proper samples to obtain a suitable decision boundary for the
open category (Scheirer et al., 2012; Liang et al., 2018; Shu et al., 2021).
In this work, building upon ideas from one-class classification (Schölkopf et
al., 2001; Ruff et al., 2018) and one-vs-all support vector machines (Rifkin
and Klautau, 2004), we propose a one-versus-rest framework via training simple
binary classifiers for each predefined category. Based on this framework, we
then introduce an effective open-sample generation approach to train these
classifiers in Section 3.3.

We build an auxiliary one-versus-rest binary classifier for each known
category, and take $m$-th category as an example to illustrate. Given a text
sequence ${\bm{x}}$, we use the BERT pretrained with classification loss as
the feature extractor $f_{\psi^{enc}}(\cdot)$ to extract features
${\bm{z}}\in{\mathbb{R}}^{d}$ to be fed to the binary classifiers, where $d$
is the dimension of the feature space. Each category is provided with a binary
classifier denoted as
$g_{\theta^{cls}_{m}}({\bm{z}}):{\mathbb{R}}^{d}\rightarrow{\mathbb{R}}$, such
that if $g_{\theta^{cls}_{m}}({\bm{z}})>0$ then the input text ${\bm{x}}$
belongs to the $m$-th category or vice versa belongs to any of the other
categories. We parameterize the entire binary classification framework for the
$m$-th category as $\theta_{m}=(\psi^{enc},\theta^{cls}_{m})$.

To learn each binary classifier $g_{\theta^{cls}_{m}}(\cdot)$ from training
data $\mathcal{D}$, we first construct a _positive_ set
$\\{{\bm{x}}_{1},{\bm{x}}_{2},\ldots,{\bm{x}}_{N_{m}}\\}$ using data points
with label $l_{m}$ from $\mathcal{D}$ and a _negative_ set
$\\{\hat{{\bm{x}}}_{1},\hat{{\bm{x}}}_{2},\ldots,\hat{{\bm{x}}}_{N-N_{m}}\\}$
by data points not in category $l_{m}$ but also from $\mathcal{D}$. The total
number of samples within category $m$ is $N_{m}$, and $N-N_{m}$ is the number
of remaining samples in $\mathcal{D}$. Each binary classifier is optimized by
minimizing the binary cross-entropy loss function $\mathcal{L}^{rest}$:

$\displaystyle\mathcal{L}^{rest}$
$\displaystyle(\theta^{cls}_{m})=\sum_{i\in[N_{m}]}\log(1+\exp(-g_{\theta_{m}}({\bm{x}}_{i})))$
$\displaystyle+$
$\displaystyle\sum_{i\in[N-N_{m}]}\log(1+\exp(g_{\theta_{m}}(\hat{{\bm{x}}}_{i}))).$
(2)

During the inference phase, we have

$\displaystyle\hat{y}=\begin{cases}\text{open,}&\text{if
}g_{\theta_{m}}({\bm{x}}_{i})<0,\forall m\in[C];\\\ \text{known},&\text{
otherwise}.\end{cases}$

We assign a testing example to the open category $l_{0}$ if it is detected as
unknown in all $C$ binary classifiers. Otherwise, we pass the example to the
known classifier to categorize the known categories. The entire inference
pipeline is illustrated in Figure 2(b).

###  3.3 Adaptive Negative Sampling (ANS)

In practice, it is problematic to learn a good binary classifier with only the
aforementioned negative samples from $\mathcal{D}$. The sample space of the
open category is complicated, considering that new-category samples could
originate from different topics and sources, relative to the known classes. So
motivated, some existing methods introduce additional external data as
negative samples.

To alleviate the issue associated with the lack of real negative samples, we
propose to synthesize negative samples $\tilde{{\bm{x}}}$. Considering that it
is hard to create actual open-category text examples, we choose to draw
virtual text examples $\tilde{{\bm{z}}}$ in the feature space (Miyato et al.,
2016; Zhu et al., 2019). Compared with the token space of text, the feature
space is typically smoother (Bengio et al., 2013), which makes it convenient
to calculate gradients (Wang et al., 2021).

For all known samples in a category $l_{m}$, points that are away from these
samples can be recognized as _negative_ to classifier
$g_{\theta_{m}^{cls}}(\cdot)$. The entire feature space ${\mathbb{R}}^{d}$
contains essentially an uncountable set of such points, among which we are
mostly interested in those near the known samples. Consequently, capturing
these points will be helpful to characterize the decision boundary.

To give a mathematical description of these points, we assume that data
usually lies on a low-dimensional manifold in a high-dimensional space (Pless
and Souvenir, 2009). The low-dimensional manifold can be viewed as a local-
Euclidean space, thus we can use the Euclidean metric to measure distances
locally for each known data ${\bm{z}}_{i}$. Under this assumption, the set of
pseudo negative samples $\mathcal{N}_{i}(r)$ for ${\bm{z}}_{i}$, which we call
_adaptive synthesized open set_ , can be described as follows,

$\displaystyle\mathcal{N}_{i}(r)=\vcentcolon\Big{\\{}\tilde{{\bm{z}}}:$
$\displaystyle
r\leq\|\tilde{{\bm{z}}}-{\bm{z}}_{j}\|_{2};\|\tilde{{\bm{z}}}-{\bm{z}}_{i}\|_{2}\leq\gamma\cdot
r,$ $\displaystyle~{}\forall j:y_{j}=m\Big{\\}},$ (3)

where $r$ is the distance radius and $\gamma>1$ is a hyperparameter. Note that
each known sample ${\bm{z}}_{i}$ has an associated adaptive synthesized open
set. As defined above, this set is subject to two inequalities. The first
keeps synthesized samples away from all known samples within category $m$. The
second implies that the synthesized samples should not be too far from the
known samples. An intuitive geometric interpretation is that when $j=i$, the
space implied by these two constraints is a spherical shell with inner radius
$r$ and outer radius $\gamma\cdot r$.

To get the radius $r$, we first calculate the covariance matrix $\Sigma$ of
${\bm{z}}$ using known samples from category $m$ and choose $r$, s.t.
$r\leq\sqrt{2\operatorname{Tr}(\Sigma)}$ and $\gamma
r\geq\sqrt{2\operatorname{Tr}(\Sigma)}$. This is under the consideration that
$\sqrt{2\operatorname{Tr}(\Sigma)}$ is the average Euclidean distance between
random samples drawn from a distribution with covariance matrix $\Sigma$.

The estimation is supported by the following proposition,

###### Proposition 1

The expectation of the euclidean distance between random points sampled from
distribution with covariance matrix $\Sigma$ is smaller than
$\sqrt{2\operatorname{Tr}(\Sigma)}$, i.e.

$\displaystyle\mathbb{E}_{{\bm{x}},{\bm{y}}\sim\mathcal{D}({\bm{\mu}},\Sigma)}\sqrt{\|{\bm{x}}-{\bm{y}}\|^{2}}\leq\sqrt{2\operatorname{Tr}{\Sigma}}$
(4)

The proof can be found in supplementary. In our experiments, we fix $\gamma=2$
and $r=8$. The choice of $r$ is relevant to the covariance matrix of the
features in representation space. The detailed justification for our selection
is provided in Appendix A.3. Ablation studies (Figure 3) show that model
performance is not very sensitive to the chosen $r$.

#### Binary Classification with ANS

According to Equation 3.3, each sample from a known category $m$ contributes
an adaptive synthesized set of open samples $\mathcal{N}_{i}(r)$. The
classifier $g_{\theta_{m}^{cls}}(\cdot)$ is expected to discriminate them as
negative. The corresponding objective function is the binary cross-entropy
loss,

$\displaystyle\mathcal{L}^{syn}(\theta^{cls}_{m})=\sum_{i\in[N_{m}]}\log(1+\exp(g_{\theta^{cls}_{m}}(\tilde{{\bm{z}}}_{i}))),$

where $\tilde{{\bm{z}}}_{i}$ is sampled from $\mathcal{N}_{i}(r)$ and $N_{m}$
is the total number of known samples with category $l_{m}$. However, there
exist uncountably many points in $\mathcal{N}_{i}(r)$. Randomly sampling one
example from $\mathcal{N}_{i}(r)$ is not effective. Alternatively, we choose
the most representative ones that are hard for the classifier to classify it
as _negative_. Consistent with this intuition, the $\max(\cdot)$ operator is
added to select the most challenging synthetic open sample distributed in
$\mathcal{N}_{i}(r)$.

$\displaystyle\mathcal{L}^{syn}(\theta^{cls}_{m})=$ (5) $\displaystyle\hskip
11.38109pt\sum_{i\in[N_{m}]}\max\limits_{\tilde{{\bm{z}}}_{i}\in\mathcal{N}_{i}(r)}\log(1+\exp(g_{\theta^{cls}_{m}}(\tilde{{\bm{z}}}_{i}))).$

Finally, the complete loss accounting for open recognition is summarized as:

$\displaystyle\mathcal{L}^{open}=\mathcal{L}^{rest}+\lambda\mathcal{L}^{syn},$
(6)

where $\lambda$ is a regularization hyperparameter.

Directly minimizing the objective function in Equation 6 subject to the
constraint in Equation 3.3 is challenging. In the experiments, we adopt the
projected gradient descend-ascend technique (Goyal et al., 2020) to solve this
problem.

% |  | BANKING | CLINC | StackOverflow
---|---|---|---|---
Methods | Accuracy | F1-score | Accuracy | F1-score | Accuracy | F1-score
25 | MSP | 43.67 | 50.09 | 47.02 | 47.62 | 28.67 | 37.85
DOC | 56.99 | 58.03 | 74.97 | 66.37 | 42.74 | 47.73
OpenMax | 49.94 | 54.14 | 68.50 | 61.99 | 40.28 | 45.98
DeepUnk | 64.21 | 61.36 | 81.43 | 71.16 | 47.84 | 52.05
ADB | 78.85 | 71.62 | 87.59 | 77.19 | 86.72 | 80.83
SelfSup* | 74.11 | 69.93 | 88.44 | 80.73 | 68.74 | 65.64
Ours | 83.93 | 76.15 | 92.64 | 84.81 | 90.88 | 84.52
50 | MSP | 59.73 | 71.18 | 62.96 | 70.41 | 52.42 | 63.01
DOC | 64.81 | 73.12 | 77.16 | 78.26 | 52.53 | 62.84
OpenMax | 65.31 | 74.24 | 80.11 | 80.56 | 60.35 | 68.18
DeepUnk | 72.73 | 77.53 | 83.35 | 82.16 | 58.98 | 68.01
ADB | 78.86 | 80.90 | 86.54 | 85.05 | 86.40 | 85.83
SelfSup* | 72.69 | 79.21 | 88.33 | 86.67 | 75.08 | 78.55
Ours | 81.97 | 83.29 | 90.23 | 88.01 | 86.08 | 85.90
75 | MSP | 75.89 | 83.60 | 74.07 | 82.38 | 72.17 | 77.95
DOC | 76.77 | 83.34 | 78.73 | 83.59 | 68.91 | 75.06
OpenMax | 77.45 | 84.07 | 76.80 | 73.16 | 74.42 | 79.78
DeepUnk | 78.52 | 84.31 | 83.71 | 86.23 | 72.33 | 78.28
ADB | 81.08 | 85.96 | 86.32 | 88.53 | 82.78 | 85.99
SelfSup* | 81.07 | 86.98 | 88.08 | 89.43 | 81.71 | 85.85
Ours | 82.49 | 86.92 | 88.96 | 89.97 | 84.40 | 87.49

Table 1:  Results of open world classification on three datasets with
different known class proportions. * indicates the use of extra datasets
during the training. The first five results of the baseline model are from
Zhang et al. (2021b). The results for SelfSup are from Zhan et al. (2021).

#### Projected Gradient Descend-Ascent

Algorithm 1 Adaptive Negative Sampling.

1:Input: Training data
$\mathcal{D}={\bm{x}}_{1},{\bm{x}}_{2},\cdots,{\bm{x}}_{n},\hat{{\bm{x}}}_{1},\cdots,\hat{{\bm{x}}}_{n}$.
Parameters of current binary classifier $\theta_{m}$.

2:Hyper-Parameters: Radius $r$, step-size $\eta$, number of gradient steps $k$

3:for Batch number $B=1,\cdots,n_{0}$ do

4: $X_{B}(\hat{X}_{B})$: Collect a batch of positive (negative) samples.

5: Calculate loss $\mathcal{L}_{1}=\mathcal{L}^{real}(X_{B},\hat{X}_{B})$
using Eq. 2

6: Calculate the feature ${\bm{z}}_{B}$ over the positive samples.

7: Adaptive Negative Sampling:

8: sample ${\bm{\epsilon}}\sim{\mathbb{N}}(0,4\cdot diag(\Sigma_{\bm{z}}))$

9: for $i=1,\cdots,k$ do

10: Calculate loss $\ell({\bm{z}}_{B}+{\bm{\epsilon}})$ using Eq. 5

11:
${\bm{\epsilon}}={\bm{\epsilon}}+\eta\frac{\nabla_{\bm{\epsilon}}\ell({\bm{z}}_{B}+{\bm{\epsilon}})}{\|\nabla_{\bm{\epsilon}}\ell({\bm{z}}_{B}+{\bm{\epsilon}})\|}$
$\triangleright$ Gradient Ascend

12: end for

13: Calculate $\alpha$ using Eq.3.3

14: ${\bm{\epsilon}}=\frac{\alpha}{\|{\bm{\epsilon}}\|}\cdot{\bm{\epsilon}}$

15: Calculate loss
$\mathcal{L}_{2}=\mathcal{L}^{syn}_{\theta^{\prime}_{m}}({\bm{z}}_{B}+{\bm{\epsilon}})$
using Eq. 5

16:
$\theta_{m}=\theta_{m}-\nabla_{\theta_{m}}(\mathcal{L}_{1}+\mathcal{L}_{2})$
$\triangleright$ Gradient Descend

17:end for

We use gradient descent to minimize the open recognition loss
$\mathcal{L}^{open}$ and gradient ascent to find the hardest synthetic
negative samples $\tilde{{\bm{z}}}$. The detailed steps are summarized in
Algorithm 1. As illustrated in Figure 2(c), the sample
$\tilde{{\bm{z}}_{i}}^{\prime}=\tilde{{\bm{z}}_{i}}+\epsilon$ directly derived
from gradient ascent (line 11 of Algorithm 1) might be out of the constraint
area $\mathcal{N}_{i}(r)$. We then project to the closest
$\tilde{{\bm{z}}_{i}}$ within the constraint such that
$\tilde{{\bm{z}}_{i}}=\arg\min_{\bm{u}}\|\tilde{{\bm{z}}_{i}}^{\prime}-{\bm{u}}\|^{2},\forall\
\ {\bm{u}}\in\mathcal{N}_{i}(r)$ (Boyd et al., 2004). Unfortunately, direct
search within $\mathcal{N}_{i}(r)$ defined in Equation 3.3 requires complex
computation over entire training data $\mathcal{D}$. Based on our assumption
that the training samples lie on a low-dimensional manifold and the empirical
observation that $\tilde{{\bm{z}}}_{i}^{\prime}$ is always closest to the
corresponding positive point ${\bm{z}}_{i}$ relative to other positive points,
$\mathcal{N}_{i}(r)$ can be further relaxed to the sphere shell around sample
${\bm{z}}_{i}$:
$\mathcal{N}_{i}(r)=\\{\tilde{{\bm{z}}}:r\leq\|\tilde{{\bm{z}}}-{\bm{z}}_{i}\|_{2}\leq\gamma\cdot
r\\}$. We can then directly find the synthetic negative sample via a
projection along the radius direction, i.e.,
$\tilde{{\bm{z}}_{i}}=\tilde{{\bm{z}}_{i}}^{\prime}+\alpha\frac{\tilde{{\bm{z}}_{i}}^{\prime}-{\bm{z}}_{i}}{\|\tilde{{\bm{z}}_{i}}^{\prime}-{\bm{z}}_{i}\|}$,
where $\alpha$ is adjusted to guarantee
$\tilde{{\bm{z}}_{i}}\in\mathcal{N}_{i}(r)$:

$\displaystyle\alpha=\begin{cases}1,&\text{if
}r\leq\|\tilde{{\bm{z}}}_{i}-{\bm{z}}_{i}\|_{2}\leq\gamma\cdot r\\\
\frac{r\gamma}{\|\tilde{{\bm{z}}}_{i}-{\bm{z}}_{i}\|_{2}},&\text{if
}\gamma\cdot r\leq\|\tilde{{\bm{z}}}_{i}-{\bm{z}}_{i}\|_{2}\\\
\frac{r}{\|\tilde{{\bm{z}}}_{i}-{\bm{z}}_{i}\|_{2}},&\text{if
}\|\tilde{{\bm{z}}}_{i}-{\bm{z}}_{i}\|_{2}\leq r\end{cases}$

In the future, we would like to consider relaxing these constraints by only
considering the nearest $k$ points instead of all the points within a
category.

##  4 Experiments

We conduct experiments on three datasets: Banking, CLINC and Stackoverflow.
Details and examples of the datasets are found in Appendix A.4.

#### Task Design

We apply the same settings as Shu et al. (2017) and Lin and Xu (2019a). For
each dataset, we sample $25\%$, $50\%$, $75\%$ categories randomly and treat
them as the _known category_. Any other categories out of the known categories
are grouped into the _open category_. In the training and validation set, only
samples within the known category are kept. All the samples in the testing set
are retained, and the label of samples belonging to open categories is set to
$l_{0}$. Importantly, samples from the open category are never exposed to the
model in the training and validation process.

#### Evaluation Metrics

The model needs to identify the samples with the open category, as well as
classify the known samples correctly. Following Shu et al. (2017); Zhang et
al. (2021b), we use accuracy and macro F1 as our evaluation metrics. The
accuracy measures the overall performance, considering that open-world
classification can be treated as a $C+1$ classification problem. F1 is a
binary classification metric mainly used for evaluating the performance of
open category recognition. The F1-score reported in this paper is the mean of
the macro F1-score per category (including the open category), where the
positive category is the corresponding one and negatives are all the other
categories. F1-known is the average over F1-score of all known categories.
F1-open is the F1-score of the open category.

#### Experimental setting

We use the BERT-base-uncased model to initialize the feature extractor
$f_{\phi_{enc}}$ and freeze the first ten layers of BERT during training. Note
that all results are the mean of ten trials with different random seeds. Other
experimental details are included in Appendix A.5.

###  4.1 Results

% | Methods | Acc | F1 | F1-open | F1-known
---|---|---|---|---|---
25 |  Baseline ($\lambda=0$) | 57.05 | 53.12 | 63.83 | 52.84
$+$ Gaussian Noise | 90.37 | 82.09 | 93.75 | 81.78
$+$ Projection | 92.02 | 83.99 | 94.91 | 83.71
$+$ Ascend (Ours) | 92.32 | 84.34 | 95.11 | 84.05
50 |  Baseline ($\lambda=0$) | 64.60 | 71.65 | 60.28 | 71.80
$+$ Gaussian Noise | 88.01 | 86.90 | 89.82 | 86.86
$+$ Projection | 90.22 | 88.18 | 92.01 | 88.12
$+$ Ascend (Ours) | 90.23 | 88.22 | 92.02 | 88.17
75 |  Baseline ($\lambda=0$) | 76.17 | 83.63 | 63.23 | 83.81
$+$ Gaussian Noise | 88.67 | 90.45 | 86.71 | 90.48
$+$ Projection | 88.89 | 89.95 | 87.52 | 89.97
$+$ Ascend (Ours) | 88.96 | 89.97 | 87.62 | 90.00

Table 2: Ablation study on the negative samples generation. The results are
conducted on CLINC with known classes proportion 25%, 50% and 75%. Baseline
represents the experiment with no synthesized negative samples under the one-
versus-rest framework.  Figure 3: Ablation study on the radius $r$ used in
adaptive negative sampling process under the setting with 50% known
categories.

Table 1 compares our approach with previous state-of-the-art methods using
accuracy and F1. Our implementation is based on Zhang et al. (2021a). The
baselines include threshold finding methods, MSP (Hendrycks and Gimpel, 2016),
DOC (Shu et al., 2017), OpenMax (Bendale and Boult, 2015), ADB (Zhang et al.,
2021b); and feature learning methods, DeepUnk (Lin and Xu, 2019a); and
negative data generation method SelfSup (Zhan et al., 2021). We did not
include results from ODIST (Shu et al., 2021) because their method relies on
MNLI-pretrained BART, which is not currently public and their model
performance drops dramatically if not coupled with ADB. Note that SelfSup uses
additional datasets, without which the accuracy on 50% CLINC drops from 88.33
to 83.12.

Our approach performs better than most previous methods, even better than the
method using additional datasets, with the greatest improvement on CLINC. This
is in accordance with SelfSup (Zhan et al., 2021), which also benefits the
most on CLINC by adding negative samples. This implies that our synthesized
negative samples are of high quality and could possibly be used as extra
datasets in other methods.

The average performance gain in these three datasets decreases as the known
category ratio increases, i.e., compared to the strongest baseline ADB, our
accuracy improvements in the three datasets are $5.08,3.11,1.42$ under the
setting of $25\%,50\%,75\%$. With more known categories available, the more
diverse the known negative samples will be, allowing the model to better
capture the boundaries of the positive known categories while reducing the
impact of synthetic samples.

The comparison with baselines on F1-open and F1-known can be found in Appendix
A.3.

Methods | Add Neg. | Acc | F1 | F1-open | F1-known
---|---|---|---|---|---
MSP |  | 44.46 | 51.95 | 43.22 | 52.41
$\checkmark$ | 57.07 | 58.92 | 61.40 | 58.79
ADB |  | 78.39 | 71.53 | 84.18 | 70.86
$\checkmark$ | 79.71 | 73.01 | 85.18 | 72.12
One-vs -Rest |  | 51.33 | 41.40 | 50.83 | 43.54
$\checkmark$ | 80.11 | 73.35 | 85.57 | 72.71

Table 3: Performance comparison when the synthesized negative data are added
to different baselines. The experiments are conducted on Banking with 25%
known categories.

Figure 4:  $t$-SNE plots of the feature extracted from the testing set of
CLINC with 50% known categories. Each panel corresponds to a one-versus-rest
classifier $g_{\theta_{m}^{cls}}$ with different known category $m$ acting as
the positive. Each panel’s lower right corner has a square that enlarges the
known positive data region to show the effectiveness of the synthesized
negative samples. (Best viewed in color).

###  4.2 Discussion

#### Synthesized Negative is Beneficial for a Variety of Structures

To investigate the contribution of the synthesized samples and the structure
of one-versus-rest, we performed experiments adding the synthesized samples to
two well-known baselines, MSP (Hendrycks and Gimpel, 2016) and ADB (Zhang et
al., 2021b) as shown in Table 3. Specifically, the C-way classifier in MSP and
ADB is replaced by a ($C+1$)-way classifier, with an extra head for the
synthesized negative samples. See Appendix A.7 for details.

We observe that performance increases on all baselines with synthesized
negative samples. The synthesized samples behave like data augmentation,
leading to better representation of positive samples.

Further, synthesized samples benefit one-versus-rest the most. The difference,
we believe, stems from the model’s flexibility on boundary learning. The open
category may contain sentences with various themes, making it difficult for a
single head of a ($C+1$)-way classifier to catch them all. This one-versus-
rest flexibility comes at the cost of more classifier parameters. However,
compared to the huge feature extractor BERT, the number of additional
parameters is relatively small. Distillation techniques can be used to build a
smaller model if necessary, for instance, where there are thousands of known
categories.

#### Adaptive Negatives Samples Generation

Our adaptive negative sample generation consists of three modules (a) adding
Gaussian noise to the original samples (line $8$ in Algorithm 1). (b) gradient
ascent (line $10\sim 11$) (c) projection (line $13\sim 14$). We add each
module to the baseline in turn to study their importance. The baseline
experiment uses the vanilla one-versus-rest framework described in Section
3.2, without the use of synthesized negative samples. Experiments are
conducted on CLINC as shown in Table 2.

The following describes our findings from each experiment: ($i$) Adding
samples with noise as negative alleviates the overconfidence problem of the
classifier and improves the results significantly. The noise level needs to be
designed carefully since small noise blurs the distinction between known and
open, while large noise is ineffective.

($ii$) Constraining synthesized samples to $\mathcal{N}(r)$ improves
performance by keeping synthesized samples from being too close or too far
away from positive known samples.

($iii$) Adding a gradient ascent step further enhances performance. The
improvement over the previous step is marginal. Our hypothesis is that the
calculated gradient could be noisy, since the noise we add is isotropic and
may be inconsistent with (outside of) the manifold of the original data.

#### Radius $r$ Analysis

In the adaptive negative sample generation process, the radius $r$ and
multiplier $\gamma$ are two hyperparameters that determine the upper and lower
bounds of the distance between the synthesized sample and the known sample. To
investigate the impact of radius, we fix $\gamma$ to 2 and increase the $r$
from $1$ to $256$. Note that $8$ is our default setting.

As illustrated in Figure 3, the performance gradually drops when the radius
$r$ increases, because the influence of the synthesized negative examples
reduces as the distance between them and the positive samples grows. When the
radius $r$ decreases, the classifier may be more likely to incorrectly
categorize positive as negative because the synthesized negative samples are
closer to the known positives, resulting in a decrease in accuracy and F1
score on the banking and CLINC datasets. However, the performance on
Stackoverflow improves. We hypothesize that there is a better data-adaptive
way to estimate the radius $r$ to improve the performance even further, for
example, using $k$ nearest neighbor instead of all the data in a category. We
leave this as interesting future direction.

In summary, we observe that the performance is affected by the radius, but
comparable results can be obtained for a wide value range. They are all better
than the vanilla one-versus-rest baseline, which lacks the generated negative
samples. The accuracy of baselines on Banking, CLINC and Stackoverflow is
$58.09$, $71.80$ and $64.58$, respectively.

#### Visualization

Figure 4 shows the $t$-SNE representation of the features extracted from the
second hidden layer of one-versus-rest classifier $g_{m}^{cls}$. Randomly
chosen three known categories, each corresponds to a one-versus-rest
classifier, yield three figures. The known positive/negative samples (blue)
are clustered because the features are extracted from a pretrained $C$-way
classifier Section 3.1. Open samples (pink) are scattered, some of which
overlap with the known positives (see the middle figure). Our synthesized
negatives work as expected; they are close to the known positives and bridge
the gap between the positive and other known categories.

##  5 Conclusions

We have introduced ANS, a pseudo open category sample generation approach for
open-world classification problems. The generation process is free from extra
datasets or prior knowledge. The synthesized samples are effective for
improving existing methods. Combined with one-versus-rest framework,
significant improvements are observed on three benchmarks. The gradient-based
negative sample generation proposed can be applied to other NLP tasks such as
out-of-scope discovery, which we leave as future work.

##  6 Limitations

First, both in terms of the number of training samples and the length of
sentences, all the datasets that we employ in this study are relatively small
and short. It may not scale well for long texts. Second, further work should
be done on the model’s ability to handle increasingly complicated data. The
training samples from three benchmark datasets might be too sample; many input
sentences even include the category descriptions, as shown in Tabel A.1.

##  7 Acknowledgement

This research was supported by DARPA, DOE, NIH, ONR and NSF.

## References

  * Bendale and Boult (2015) Abhijit Bendale and Terrance Boult. 2015.  Towards open world recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1893–1902.
  * Bendale and Boult (2016) Abhijit Bendale and Terrance E Boult. 2016.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1563–1572.
  * Bengio et al. (2013) Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. 2013.  Better mixing via deep representations.  In _International conference on machine learning_ , pages 552–560. PMLR.
  * Boyd et al. (2004) Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. 2004.  _Convex optimization_.  Cambridge university press.
  * Breunig et al. (2000) Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. 2000.  Lof: identifying density-based local outliers.  In _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_ , pages 93–104.
  * Casanueva et al. (2020) Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  _arXiv preprint arXiv:2003.04807_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Goyal et al. (2020) Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. 2020.  Drocc: Deep robust one-class classification.  In _International Conference on Machine Learning_ , pages 3711–3721. PMLR.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , pages 1321–1330. PMLR.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  _arXiv preprint arXiv:1909.02027_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _ICLR_.
  * Lin and Xu (2019a) Ting-En Lin and Hua Xu. 2019a.  Deep unknown intent detection with margin loss.  _arXiv preprint arXiv:1906.00434_.
  * Lin and Xu (2019b) Ting-En Lin and Hua Xu. 2019b.  A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier.  _Knowledge-Based Systems_ , 186:104979.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018.  Towards deep learning models resistant to adversarial attacks.  _ArXiv_ , abs/1706.06083.
  * Miyato et al. (2016) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016.  Adversarial training methods for semi-supervised text classification.  _arXiv preprint arXiv:1605.07725_.
  * Pless and Souvenir (2009) Robert Pless and Richard Souvenir. 2009.  A survey of manifold learning for images.  _IPSJ Transactions on Computer Vision and Applications_ , 1:83–94.
  * Rifkin and Klautau (2004) Ryan Rifkin and Aldebaro Klautau. 2004.  In defense of one-vs-all classification.  _The Journal of Machine Learning Research_ , 5:101–141.
  * Ruff et al. (2018) Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. 2018.  Deep one-class classification.  In _International conference on machine learning_ , pages 4393–4402. PMLR.
  * Scheirer et al. (2012) Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. 2012.  Toward open set recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(7):1757–1772.
  * Scheirer et al. (2014) Walter J Scheirer, Lalit P Jain, and Terrance E Boult. 2014.  Probability models for open set recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(11):2317–2324.
  * Schölkopf et al. (2001) Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001.  Estimating the support of a high-dimensional distribution.  _Neural computation_ , 13(7):1443–1471.
  * Shu et al. (2021) Lei Shu, Yassine Benajiba, Saab Mansour, and Yi Zhang. 2021.  Odist: Open world classification via distributionally shifted instances.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3751–3756.
  * Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017.  Doc: Deep open classification of text documents.  _arXiv preprint arXiv:1709.08716_.
  * Wang et al. (2018) Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018.  Cosface: Large margin cosine loss for deep face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5265–5274.
  * Wang et al. (2021) Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. 2021.  Energy-based open-world uncertainty modeling for confidence calibration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9302–9311.
  * Xu et al. (2015) Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015.  Short text clustering via convolutional neural networks.  In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_ , pages 62–69.
  * Yan et al. (2020) Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert YS Lam. 2020.  Unknown intent detection using gaussian mixture model with an application to zero-shot intent classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1050–1060.
  * Zeng et al. (2021) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021.  Modeling discriminative representations for out-of-domain detection with supervised contrastive learning.  _arXiv preprint arXiv:2105.14289_.
  * Zhan et al. (2021) Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert Lam. 2021.  Out-of-scope intent detection with self-supervision and discriminative training.  _arXiv preprint arXiv:2106.08616_.
  * Zhang et al. (2021a) Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021a.  TEXTOIR: An integrated and visualized platform for text open intent recognition.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 167–174.
  * Zhang et al. (2021b) Hanlei Zhang, Hua Xu, and Ting-En Lin. 2021b.  Deep open intent classification with adaptive decision boundary.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14374–14382.
  * Zhou et al. (2022) Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022.  Knn-contrastive learning for out-of-domain intent classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5129–5141.
  * Zhu et al. (2019) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019.  Freelb: Enhanced adversarial training for natural language understanding.  _ICLR_.

##  Appendix A Appendix

###  A.1 Ethical Consideration

The topics of the three datasets we use in this paper are relatively simple,
covering only the information needed for classification (check Table A.1). The
category labels are either everyday intentions or technical terms in computer
science. There are no potentially sensitive topics or contents that we are
aware of. All three datasets have been published and are included in our
appendix.

###  A.2 Related work: Adversary Augmentation

The gradient descend-ascend technique has been used successfully in
adversarial attacks (Madry et al., 2018; Zeng et al., 2021); however, it
differs from ours in terms of motivation and loss formulation.

Madry et al. (2018) sought for samples that were similar to the training
sample but had a substantial loss given the paired label. The addition of
generated samples during training may strengthen the model’s resistance to
adversarial attacks by avoiding inputs that are indistinguishable from genuine
data but improperly categorised. The associated optimization formula is

$\displaystyle\min_{\theta}\ \mathbb{E}_{{\bm{x}},y\sim D}[\max_{\delta\in
S}l(\theta,{\bm{x}}+\delta,y)],$ (7)

where $y,{\bm{x}}$ are the training data, and $l$ could be any classification
model parameterized by $\theta$. $S$ is the an adversarial perturbation
$l_{\infty}$ ball.

Our work targets on shrinking the decision boundary. We need to treat the
samples with positive labels in a specific region $\mathcal{N}(r)$(defined in
Equation 3.3) as negative, i.e.

$\displaystyle\min_{\theta}\ \mathbb{E}_{{\bm{z}}\sim
D}[\max_{\delta\in\mathcal{N}(r)}l(\theta,{\bm{z}}+\delta,-1)],$ (8)

where ${\bm{z}}$ is the positive sample from dataset $D$, $l$ is a binary
classifier with parameter $\delta$. This equation behaves the same as Equation
5.

###  A.3 Explanation on Radius $r$

###### Proposition 2

The expectation of the euclidean distance between random points sampled from
distribution with covariance matrix $\Sigma$ is smaller than
$\sqrt{2\operatorname{Tr}(\Sigma)}$, i.e.

$\displaystyle\mathbb{E}_{{\bm{x}},{\bm{y}}\sim\mathcal{D}({\bm{\mu}},\Sigma)}\sqrt{\|{\bm{x}}-{\bm{y}}\|^{2}}\leq\sqrt{2\operatorname{Tr}{\Sigma}}$
(9)

Proof: Given that we are measuring the distance between samples drawn from the
same distribution, we could subtract a constant value from both variables and
assume that the distribution’s expectation is zero. If ${\bm{x}}$ and
${\bm{y}}$ are random variables independently sampled from distribution with
covariance matrix $\Sigma$ and zero mean, we could have:

$\displaystyle\mathbb{E}(\|{\bm{x}}-{\bm{y}}\|^{2}_{2})$
$\displaystyle=\sum_{i}\mathbb{E}({\bm{x}}_{i}^{2}-2{\bm{x}}_{i}{\bm{y}}_{i}+{\bm{y}}_{i}^{2})$
$\displaystyle=2\sum_{i}(\mathbb{E}({\bm{x}}_{i}^{2})-\mathbb{E}({\bm{x}}_{i})\mathbb{E}({\bm{y}}_{i}))$
$\displaystyle=2\sum_{i}(\mathbb{E}({\bm{x}}_{i}^{2})$
$\displaystyle=2\operatorname{Tr}(\Sigma)$

For a random variable $Z$, Jensen’s inequality gives us

$\displaystyle\mathbb{E}(\sqrt{Z})\leq\sqrt{\mathbb{E}[Z]}$ (10)

The combination of the two equations above proves the proposition,

$\displaystyle\mathbb{E}(\|{\bm{x}}-{\bm{y}}\|_{2})\leq\sqrt{2\operatorname{Tr}(\Sigma)}$

In experiment, we choose the mean of the last layer of BERT as the latent
representation ${\bm{z}}\in\mathbb{R}^{768}$. When calculating the trace, only
the variance of each dimension, are required. On three datasets, the predicted
distance per category falls primarily into $[8,12]$, we fix $r=8$ for all the
experiments.

For each positive point, we could sample several adaptive negative samples.
The distance between the synthesized negative and the chosen positive is
determined by $r$. Meanwhile, we can also calculate the distance between the
synthesized negative and other positive known samples.

We find that even when the radius is set to be less than the average distance,
the synthesized negative samples have a much greater distance to other known
points. In theory, known samples are on a low-dimensional manifold, whereas
synthesized points are in a high-dimensional space, and the probability of
sampled points falling into the manifold is zero. We calculated the distance
between the synthesized sample and other known samples in the same category
empirically, and discovered that their distances are nearly twice the average
distance.

A further ablation study on different options of the radius can be found in
the main context, where we have comparisons over different radius, i.e.,
$r\in\\{1,4,8,16,64\\}$.

Dataset | Category | Examples
---|---|---
Banking | exchange_via_app |  | What currences are available for exchange?
---
Does your app allow currency exchange from USD to GBP?
I want to exchange USD and GBP with the app
| wrong_exchange_
---
rate_for_cash
_withdrawal
| I was given the wrong exchange rate when getting cash
---
I think I was charged a different exchange rate than what was posted at the
time.
I need an accurate exchange rate, when I make my withdrawals.
| card_payment_
---
wrong_exchange_rate
| Why didn’t I receive the correct exchange rate for an item that I purchased?
---
The fee charged when I changed rubles into British pounds was too much.
I am being charged the wrong amount on my card.
CLINC | flight_status |  | what time is this flight supposed to land
---
what time will i be able to board the plane
so when is my flight landing
time |  | what time is it in adelaide, australia right now
---
please tell me the time
how late is it now in ourense
how_busy |  | how long will i wait for a table at red lobster
---
can i expect chili’s to be busy at 4:30
so how busy is the outback steakhouse at 5 pm
Stackoverflow | wordpress |  | How Display Recent Posts in all 3 languages at once in Wordpress
---
How secure is Wordpress?
Need MySQL Queries to delete WordPress Posts and Post Meta more than X Days
Old
apache |  | Apache / PHP Disable Cookies for Subdomain ?
---
Increase PHP Memory limit (Apache, Drupal6)
is setting the uploads folder 777 permision secure
excel |  | Condition to check whether cell is readonly in EXCEL using C#
---
EXCEL XOR multiple bits
How I can export a datatable to excel 2007 and pdf from asp.net?

Table A.1: Extracted samples from three main datasets.

Dataset |  | Avg. Samples
---
per category
Avg. Length | Classes
Banking | 117 | 11.91 | 77
CLINIC | 100 | 8.31 | 150
StackOverflow | 600 | 9.18 | 20

Table A.2: Statistics of benchmark datasets

###  A.4 Dataset

All three datasets are in English. The label distributions of each dataset are
balanced. Banking (Casanueva et al., 2020) is a data set for intent detection
in the banking domain with 77 fine-grained categories. It contains 13,083
samples in total. We follow the original splitting and sampling, i.e., 9,003,
1,000, and 3,080 for training, validation and testing respectively.

CLINC (Larson et al., 2019) is an intent classification dataset originally
designed for out-of-scope detection. It owns 22,500 in-scope samples and 1,200
out-of-scope samples that existed only in the testing phase. The divisions on
the in-scope ones follow the original training, validation and testing
splitting, i.e., 15,000, 3,000 and 5,700.

StackOverflow (Xu et al., 2015) consists of 20 categories of technical
question titles. We utilize the prepossessed version with 12,000 for training,
2,000 for validation and 6,000 for testing.

The statistics of the datasets are summarized in Table A.2. We also provide
raw data samples of each dataset in Table A.1 for an intuitive understanding
of the open world recognition task.

###  A.5 Experimental Details

All experiments are executed on a single NVIDIA Titan Xp GPU with 12,196M
memory. All the experiments are done on NVIDIA X

#### Known Category Classification

The classifier $f_{\psi_{cls}}$ is a fully connected neural network composed
of one hidden layer with ReLU activation function. The hidden size is 768.

The learning rate of the transformer part and non-transformer part are
${5}\mathrm{e}{-}5$ and ${1}\mathrm{e}{-}4$ respectively. The total number of
training epochs is 100. Learning rate decay and early stops are applied.

Training on this part takes about 10-20 minutes, depending on when the early
stopping is triggered.

#### one-versus-rest Binary Classification

During the training of one-versus-rest structure, we fix the parameters of the
feature extractor $\psi^{enc}$.

For the one-versus-rest module, the feature ${\bm{z}}$ is chosen as the mean
of the output of the BERT model’s last layer. The classifier is a fully
connected three-layer neural network with ReLU as the activation function. The
numbers of hidden neurons are (256,64), respectively. Dropout is added per
hidden layer. The learning rate of the classifier is ${1}\mathrm{e}{-}3$ for
Banking and CLINC, ${3}\mathrm{e}{-}4$ for Stackoverflow. The total number of
epochs for each classifier is $\min(C,20)$ to avoid overfitting. $\gamma$ is
$0.5$.

Currently, we train each one-versus-rest classifier individually, and this
takes roughly a minute per head. As a result, the total time grows linearly
with the number of known categories. Parallel training of multiple heads can
increase efficiency if necessary.

#### Model Size

The parameters of the BERT backbone model and the ovr classifier are
approximately 109 million and 0.2 million, respectively, implying that the
maximum number of parameters from one-versus-rest is only about one-fifth of
BERT (75% CLINC).

#### Reproducibility Checklist: hyper-parameter search

We didn’t include results from the validation set considering there is a huge
gap between the current validation set and test set; test set contains open
category samples while the validation set does not.

It is difficult to study the hyper-parameter setting because we lack an
effective validation set with open category samples and the testing set is
unavailable during training. To solve this, we construct a "pseudo dataset" by
selecting a subset as "sub-known" from all known categories and treating
others as "sub-open". Taking 50% CLINC as an example, we take a quarter of the
known category as "sub-known" and the others as "sub-open". We discover that
the rules we developed using these synthesized datasets can be transferred to
formal experiments. We choose the proper hyper-parameter according to F1.

The hyperparameters we manually tried include training epoch ($10$, $20$),
learning rate ($1e-3$,$1e-4$ for the classifier head, $1e-4$, $5e-5$, $1e-5$
for BERT, note that the learning rate of the classifier head is always larger
than BERT). The ablation study and the analysis on radius $r$ can be found in
Figure 3 and Appendix A.3. The hyper-parameters in gradient ascend are not
sensitive to the final experiments.

###  A.6 More results

#### F1-known and F1-open

|  | BANKING | CLINIC | StackOverflow
---|---|---|---|---
% | Methods | Open | Known | Open | Known | Open | Known
25 | MSP | 41.43 | 50.55 | 50.88 | 47.53 | 13.03 | 42.82
DOC | 61.42 | 57.85 | 81.98 | 65.96 | 41.25 | 49.02
OpenMax | 51.32 | 54.28 | 75.76 | 61.62 | 36.41 | 47.89
DeepUnk | 70.44 | 60.88 | 87.33 | 70.73 | 49.29 | 52.60
ADB | 84.56 | 70.94 | 91.84 | 76.80 | 90.88 | 78.82
SelfSup* | 80.12 | 69.39 | 92.35 | 80.43 | 74.86 | 63.80
Ours | 86.39 | 72.29 | 95.33 | 84.53 | 93.96 | 82.63
50 | MSP | 41.19 | 71.97 | 57.62 | 70.58 | 23.99 | 66.91
DOC | 55.14 | 73.59 | 79.00 | 78.25 | 25.44 | 66.58
OpenMax | 54.33 | 74.76 | 81.89 | 80.54 | 45.00 | 70.49
DeepUnk | 69.53 | 77.74 | 85.85 | 82.11 | 43.01 | 70.51
ADB | 78.44 | 80.96 | 88.65 | 85.00 | 87.34 | 85.68
SelfSup* | 67.26 | 79.52 | 90.30 | 86.54 | 71.88 | 79.22
Ours | 81.06 | 83.52 | 92.16 | 87.95 | 86.83 | 85.81
75 | MSP | 39.23 | 84.36 | 59.08 | 82.59 | 33.96 | 80.88
DOC | 50.60 | 83.91 | 72.87 | 83.69 | 16.76 | 78.95
OpenMax | 50.85 | 84.64 | 76.35 | 73.13 | 44.87 | 82.11
DeepUnk | 58.54 | 84.75 | 81.15 | 86.27 | 37.59 | 81.00
ADB | 66.47 | 86.29 | 83.92 | 88.58 | 73.86 | 86.80
SelfSup* | 60.71 | 87.47 | 86.28 | 89.46 | 65.44 | 87.22
Ours | 70.54 | 88.13 | 87.20 | 89.18 | 74.82 | 88.34

Table A.3: Macro-F1 score on known category and open category of open world
classification on three datasets with different known class proportions. *
means extra datasets are used during the training. This table complements to
Table 1

The definition of F1-known and F1-open can be found in Section 4. Table A.3
shows the comparisons between the baselines and ours.

#### Reproducibility Checklist: Differences on Datasets

During this process, we found that dataset CLINC is the robustest to change of
hyper-parameters while dataset Stackoverflow is the weakest. A similar
observation also shows in Zeng et al. (2021). It works the best on CLINC and
worst on Stackoverflow.

We hypothesize that the difference comes from the quality of the raw data. As
shown in Table Table A.1, the category of the input in Stackoverflow is
usually included in the original sentence and we name them “easy”. Rare
sentences do not follow to this rule and we call them “hard”. This leads to an
observation in empirical experiments that the number of training epochs should
be controlled in a limited range; otherwise many open category samples would
be wrongly categorized to the known category.

This is in consistent with the finding in noisy labeling classification. The
neural network will first fit the clean label before overfitting the noisy
labeled samples. Under the current setting, the “hard” corresponds to the
noisy sample. The one-versus-rest will first fit the easier one, followed by
the harder one. When the harder one is classified correctly, many open
categories could also be classified into this known category.

ADB (Zhang et al., 2021b) avoids this problem by working directly on the pre-
trained features. It can statistically filter out the influence of noisy
samples. Though ADB does not require extra hyper-parameter tuning, we found
that the position of features extracted from the model has an impact on the
final performances.

In summary, the differences between datasets are an intriguing topic that
merits further investigation in the future.

#### Reproducibility Checklist: Standard Deviation

| % | Accuracy | F1 | F1-open | F1-known
---|---|---|---|---|---
Banking | 25 | 1.83 | 1.76 | 1.47 | 1.78
50 | 1.05 | 1.00 | 1.14 | 1.00
75 | 1.05 | 0.69 | 2.45 | 0.67
CLINC | 25 | 1.85 | 2.60 | 1.25 | 2.64
50 | 0.86 | 0.74 | 0.75 | 0.74
75 | 0.68 | 0.43 | 0.91 | 0.42
StackOverflow | 25 | 1.48 | 2.01 | 1.02 | 2.22
50 | 2.85 | 2.31 | 3.13 | 2.25
75 | 1.21 | 0.54 | 3.10 | 0.43

Table A.4: Standard deviation of results collected from different random
seeds. This table complements to Table 1

As shown in Table A.4, larger known category ratios are more likely to be
associated with lower variance; this is to be expected because more samples
make the training more stable.

###  A.7 Ablation Study: Synthesized Negative Samples on other Structures

Negative sample generation for MSP and ADB follow the same process as ours,
except that the gradient ascend is removed. The complete version is left for
future work.

#### MSP with negative sampling

The original MSP is a C-way classifier $f^{cls}$ trained with cross-entropy
loss during the training. In inference, the confidence score
$p({\bm{x}}_{i})=\text{Softmax}(f^{cls}({\bm{x}}_{i}))$ is first calculated.
If the category with maximum probability $p_{m}$ is lower than $0.5$, the
corresponding input is recognized as “open” category. Otherwise, this sample
belongs to the category $m$, i.e.,

$\displaystyle\hat{y}=\begin{cases}\text{open,}&\text{if
}\max(p({\bm{x}}_{i}))<0.5\\\ \arg\max p({\bm{x}}_{i}),&\text{
otherwise}.\end{cases}$

In MSP with negative settings, an extra category $l_{0}$ is added for
synthesized negatives. The inference now becomes

$\displaystyle\hat{y}=\begin{cases}\text{open,}&\text{if
}\max(p({\bm{x}}_{i}))<0.5\\\ \text{open,}&\text{if
}\operatorname*{arg\,max}(p({\bm{x}}_{i}))==l_{0}\\\ \arg\max
p({\bm{x}}_{i}),&\text{ otherwise}\\\ \end{cases}$

Note that our MSP with synthesized negatives differs from (Zhan et al., 2021)
in two aspects, ($i$), different ways to choose the negative samples. $ii$),
their work added synthesized negative samples to the validation set, while
ours uses the origin validation set.

#### ADB with negative sampling

ADB training has two steps. The first step is to learn a good feature
extractor using C-way classifier. The second step is to learn the boundary of
each category in the pre-trained feature space.

Our modification is the first step. We replace the origin classifier with a
$C+1$-way classifier. The extra head is designed for the synthesized negative
samples. The inference step is kept the same as the original method.


# Supervised GAN Watermarking
for Intellectual Property Protection

Jianwei Fei  Nanjing University of Information Science and Technology
Nanjing, China  Zhihua Xia 2  Zhihua Xia is the corresponding author. Jinan
University
Guangzhou, China  Benedetta Tondi  University of Siena
Siena, Italy  Mauro Barni  University of Siena
Siena, Italy

###### Abstract

We propose a watermarking method for protecting the Intellectual Property (IP)
of Generative Adversarial Networks (GANs). The aim is to watermark the GAN
model so that any image generated by the GAN contains an invisible watermark
(signature), whose presence inside the image can be checked at a later stage
for ownership verification. To achieve this goal, a pre-trained CNN
watermarking decoding block is inserted at the output of the generator. The
generator loss is then modified by including a watermark loss term, to ensure
that the prescribed watermark can be extracted from the generated images. The
watermark is embedded via fine-tuning, with reduced time complexity. Results
show that our method can effectively embed an invisible watermark inside the
generated images. Moreover, our method is a general one and can work with
different GAN architectures, different tasks, and different resolutions of the
output image. We also demonstrate the good robustness performance of the
embedded watermark against several post-processing, among them, JPEG
compression, noise addition, blurring, and color transformations.

###### Index Terms:

Intellectual Property Protection, Generative Adversarial Networks, DNN
Watermarking, Security of Deep Learning

##  I Introduction

In recent years, Artificial Intelligence techniques based on Deep Learning
(DL) have made astonishing advances. For this reason, Deep Neural Network
(DNN) models are increasingly deployed in commercial products of any kind.
However, designing and training a DNN model requires very large amounts of
training data, massive computing resources, and specialized knowledge, with
development costs that are not easily sustained by individuals and even by
small companies. It is possible, then, that malicious users steal a trained
DNN model to offer unauthorized paid services, this infringing the
Intellectual Property Rights (IPR) of the true owner.

Watermarking has been applied for digital media ownership verification by
embedding digital watermarks into the cover media to be protected. The owner
is able to prove media ownership by extracting the watermark from it. Inspired
by this, some works have been proposed to protect the IPR of DNN models by
means of watermarking [1, 2, 3]. Different from media watermarking, in DNN
watermarking, the watermark is directly or indirectly embedded into the model
parameters. Based on the information required for watermark extraction, DNN
watermarking methods can be divided into two categories, generally referred to
as white-box and black-box watermarking. In white-box DNN watermarking, access
to the internal parameters of the model is required for watermark extraction.
The watermark is embedded into the weights or the activations of the network.
Embedding is typically performed by defining a proper loss function for the
optimization, that includes a watermark loss term [1, 4, 5]. However, many DNN
applications are provided according to the Machine Learning as a Service
paradigm, whereby only an Application Programming Interface (API) is
available. In this case, one can not access the internal parameters of the
model to extract the watermark. This limits the practicability of white-box
DNN watermarking. In black-box DNN watermarking, the watermark is read by
checking the output of the network in correspondence to specific triggering
signals, thus avoiding the need to access the internal parameters of the model
[3].

So far, DNN watermarking has been mostly applied to Convolutional Neural
Networks (CNN). The watermarking of Generative Adversarial Networks (GAN) has
received considerably less attention. The basic observation we rely on is that
the output of GANs is so rich in entropy, that the watermark may be
retrievable from every output, rather than in correspondence of some specific
inputs only. In the case of image generation GANs, this is equivalent to
training the GAN in such a way that all the images produced by the GAN are
watermarked, thus achieving GAN watermarking by means of image watermarking.
To the best of our knowledge, only one attempt has been made in this direction
in [6], where a very simple approach is proposed that performs GAN
watermarking with reasonably good bit accuracy by simply training the GAN on a
dataset of watermarked images.

In this paper, a supervised method for GAN watermarking is proposed. We first
train a deep learning-based image watermarking network that injects an
invisible watermark into an image by means of an encoder-decoder network. Once
the watermarking network is trained, the decoding part of the network is
frozen and exploited during the training of the GAN to enforce the injection
of the watermark inside the images generated by the GAN. The generator is then
optimized by minimizing a combined loss, consisting of the original GAN loss
and a watermark loss term. The supervision performed by the watermark decoder
ensures the presence of the owner’s watermark (signature) in the generated
images. During the verification phase, the owner or third-party authority can
extract the watermark from the images using the same watermark decoder, and
match it with the owner’s watermark, to verify if they come from a certain GAN
model. We show that, with the proposed approach, the watermark can be
successfully embedded via fine-tuning an already trained GAN, for a few
thousand iterations, thus reducing the computational burden and costs of the
watermarking process. Moreover, by performing augmentation during GAN
training, good watermark robustness against common image processing operations
can be achieved.

Our contributions can be summarized as follows:

  * •

We propose a novel solution for GAN IPR protection. IPR protection is achieved
by training the GAN in a supervised manner so that any generated image
contains a prescribed invisible watermark. The watermark can be used for
ownership verification, in order to prevent GAN property rights from being
violated.

  * •

Our method is a general one and can be used to watermark different GAN
architectures, aiming at different tasks, and with different resolutions of
the generated images.

  * •

By performing augmentation of the GAN-generated images before watermark
decoding during training (that is, adding a processing layer before the
decoder network), the embedded watermark is robust against common post-
processing operations that the generated image may undergo at a later stage.
Specifically, the watermark bit accuracy remains above 75% also under very
heavy perturbations, which would make the image unusable.

  * •

Robust GAN watermarking is performed via fine-tuning an already trained GAN,
thus allowing for ownership protection with a reduced computational burden.

The rest of the paper is organized as follows: we review the main literature
dealing with DNNs and watermarking in Section II. The proposed method is
described in Section III. The experimental setting and results are reported in
Sec. IV. We conclude the paper in Section V with some remarks and hints for
future research.

##  II Related Work and Background

In this section, we review the previous works on DNN-based image watermarking
and GAN watermarking.

###  II-A DNN-based Image Watermarking

Image watermarking is a technique that has been widely used in the past for
image authentication and ownership verification. Many model-based approaches
have been proposed for watermark embedding, either in the spatial domain [7,
8] or in the frequency domain [9, 10, 11]. More recently, several data-driven
approaches, notably, methods based on deep learning, have been proposed for
image watermarking.

In DNN-based image watermarking, watermark embedding and extraction are
carried out by properly trained CNNs [12, 13]. The encoder takes an image and
a watermark message as input and generates a watermarked image from which the
decoder tries to extract the message. The two networks are trained jointly in
an end-to-end way to minimize the perturbation introduced in the watermarked
images while maximizing the bit extraction accuracy. In particular, the
watermarking method described in [12], which is the one considered in this
work to implement the watermark decoding network, hereafter referred to as
StegaStamp, can get excellent performance in terms of bit accuracy and image
quality.

###  II-B DNN Watermarking and Watermarking of Generative Models

The goal of DNN watermarking is to embed the watermark into a DNN model to
protect the IPR associated with the DNN and possibly identify illegitimate
usages. Although DNN watermarking inherits some basic concepts and methods
from classical media watermarking [14], embedding a watermark into a DNN and
recovering it from the watermarked model is quite a different piece of work
with respect to media watermarking, calling for the development of new
techniques tailored for this application scenario. Several works have been
proposed dealing with white-box and black-box watermarking. The interested
reader may refer to [15].

Figure 1: Overview of our supervised GAN watermarking method. A watermarking
network is first trained for watermark encoding and decoding . The decoder is
then used for training the watermarked GAN. The black arrows denote the path
of forwarding propagation and the red arrows denote that of backpropagation,
MSE and BCE stand for Mean Squared Error, and Binary Cross Entropy loss,
respectively.

Most of the DNN watermarking methods proposed so far focus on the protection
of discriminative networks. Watermarking a generative model for IPR protection
is a different piece of work since it requires that the watermark can be
retrieved from the output, and then used for model authentication. While in
DNN watermarking the watermark is injected inside the model itself or
associated with the behavior of the network in correspondence to some specific
inputs, with generative models, the watermark can also be injected directly in
the GAN output, namely the generated images, so that the ownership of the
images, that is the GAN model that generated them, can be established.

Arguably, training a watermarked GAN model is a more complex task than
training a standard DNN, given that the instability issue that typically
affects GANs’ training is exacerbated by the additional watermark embedding
task. Few methods performing GAN watermarking have been proposed in the last
couple of years. Most of them focus on watermarking a GAN model by embedding a
watermark inside the GAN network, as done for discriminative models. Ong et
al. [16] proposed a backdoor-based GAN protection framework. With this scheme,
the watermark is embedded only when the GAN is queried with certain inputs,
and the embedded watermark is not invisible. In [17], a method is proposed to
protect GANs by watermarking their outputs, focusing on the specific scenario
of GAN image translation. To the best of our knowledge, the only method that
aims at protecting the IPR of GANs by instructing the GAN itself to add an
invisible watermark in its generated images is [6]. This method works as
follows: first, a given invisible watermark is embedded into the training data
by exploiting a pre-trained network for image watermarking, then the
watermarked data are used to train the GAN. In this way, the invisible
watermark is automatically embedded into the images generated by the GAN.
Although the method has been proposed as a proactive solution for deepfake
forensics and deepfake attribution, it can also be regarded as a solution for
IPR protection of GAN models.

Different from [6], with our method, watermark embedding is achieved by
training the GAN in a supervised manner, that is, using the pre-trained
watermark decoder to guide the training and adding a proper watermark loss
term in the optimization. Thanks to the supervision, our method has direct
control over the bit accuracy.

##  III Supervised GAN watermarking

In this section, we describe the proposed supervised scheme for GAN
watermarking. A GAN consists of a generator $G$ and a discriminator $D$. Our
goal is to watermark $G$ in a way that a bit string, namely the watermark
message, is embedded into any image generated by $G$. A GAN model learns to
generate realistic images $G(z)$ from random noise $z$ for a given task. Our
goal is to force $G(z)$, to contain also the watermark information, so that,
when a specific decoding function $D_{w}$ is applied to $G(z)$, the decoding
function outputs $w$, where $w\in\\{0,1\\}^{N}$ is the watermark string of
length $N$ bits, associated to the model owner.

Fig. 1 illustrates the overall pipeline of our method. We leverage a pre-
trained watermarking decoder, namely $D_{w}$, that is included at the bottom
of the generator network. The output of the watermark decoder is used to guide
the GAN optimization during training via backpropagation of a proper loss. The
purpose of the (optional) processing layer is to apply some processing to the
image before decoding, in order to make the embedded watermark more robust
against post-processing. We observe that this is a very different approach
from the one considered in [6], where the watermark is embedded into the
training data and the GAN is trained as usual. In our method, the GAN itself
learns to embed the watermark during training, in addition to generating the
images in the desired domain, the training dataset remains the same.

###  III-A Watermarking network

The watermarking network is described by an encoder-decoder architecture. We
denote the encoder and decoder network as $E_{w}$ and $D_{w}$ respectively.
The output of $E_{w}$ is the watermarked image, while the output of $D_{w}$ is
the watermark message. The goal of the watermarking network training stage is
to learn a proper encoder and decoder, by minimizing the following loss:

$L_{w}=\sum_{i=1}^{n}\text{MSE}(x_{i},E_{w}(x_{i},w))+\lambda\text{BCE}(w,D_{w}(x_{w,i})),$
(1)

where $x_{i}$ denotes the input image, $x_{w,i}$ is the watermarked image,
obtained at the output of the encoder, and $n$ is the number of training
images. The first term in (1) aims to minimize the distortion between $x$ and
$x_{w}$, while the second term aims to minimize the error between the
prescribed watermark and the watermark predicted by $D_{w}$. Finally,
$\lambda$ is a parameter that weights the importance of the decoder loss in
the total loss, trading off between having a large image perturbation on one
side and a few watermark decoding errors on the other. Once the watermarking
network is trained, the decoder is frozen and used for GAN watermarking
training. Although any watermarking encoder-decoder network can be used, we
follow [6] and adopt the StegaStamp [12] network. 111It is proper to stress
that the $D_{w}$ can be any decoding function/algorithm and it does not need
to be a network, as it is only used to guide the update of the generator.

###  III-B GAN watermarking network

For a non-watermarked GAN, at each iteration of GAN training, a random noise
$z\sim P(z)$ is sampled and mapped by $G$ into an image belonging to the
target domain. On the other hand, $D$ tries to differentiate fake images
$G(z)$ and real images $x$. The discriminator network is optimized by using
the following loss

$\displaystyle L_{D}(D,G)=-\mathbb{E}_{x\sim p_{\text{data}}(x)}[\log D(x)]$
(2) $\displaystyle\hskip 85.35826pt-\mathbb{E}_{z\sim P(z)}[\log(1-D(G(z)))],$

where $p_{\text{data}}$ denotes the real data distribution, while the
generator loss is given by

$\displaystyle L_{G}(D,G)=\mathbb{E}_{z\sim P(z)}[\log(1-D(G(z)))].$ (3)

As shown in Fig. 1, in our supervised training approach, given a watermark
$w_{gt}$ (the bit string uniquely associated with the GAN trainer/owner), the
generated image is taken as input to $D_{w}$, for watermark prediction. The
predicted watermark is compared with $w_{gt}$ and a watermark loss term is
evaluated. Specifically, we consider the BCE between the predicted watermark
and $w_{gt}$. The watermark loss is then backpropagated through the generator
network in the $G$ update step, together with the standard generator loss.
Formally, the following loss is considered to train the generator of the GAN:

$\displaystyle L_{G}^{w}(D,G)=L_{G}(D,G)+\gamma\mathbb{E}_{z\sim
P(z)}[BCE(D_{w}(G(z)),w_{gt})],$ (4)

where $\gamma$ is a parameter balancing the importance of the watermark
decoding error probability in the total loss. The case $\gamma=0$ corresponds
to a conventional GAN.

We experimentally verified that, with the proposed method, the GAN can learn
to embed the watermark also when GAN watermarking is performed by fine-tuning
a pre-trained GAN model. In particular, only thousands of fine-tuning steps
are required. This greatly reduces the computational burden and allows GAN
owners to protect already trained GANs with a little cost, which is a
peculiarity and a noticeable strength of our method. Note that the
architecture illustrated in Fig. 1 is a general one, as our approach can work
with any GAN architecture. Furthermore, even if in the discussion and in the
experimental analysis we focus on the case of GAN models generating images
from noise samples (e.g., StyleGAN [18, 19], PGGAN [20]), in principle, the
same scheme can be applied to image transfer architectures (e.g.CycleGAN [21],
pix2pix [22]).

##  IV Experiments

###  IV-A Experimental Setting

To evaluate the effectiveness of our watermarking method, we applied it to
various GAN architectures, including BEGAN, PGGAN, and StyleGAN2. Each GAN
model is first trained as usual, without watermark embedding, i.e. by letting
$\gamma=0$ (warm up stage), then it is fine-tuned by adding the watermarking
loss, i.e., considering $\gamma\neq 0$. The exact details of the training
procedure are provided in Table I.

TABLE I: Parameters setting for different architectures Model | batch size | warm-up iter | fine-tune iter | $\gamma$
---|---|---|---|---
BEGAN | 64 | 400k | 3k | 0.03
PGGAN | 16 | 360k | 1k | 3
StyleGAN2 | 64 | 200k | 1k | 3

As for the training datasets of real images, we consider 200k CelebA [23] at
128$\times$128, 70k FFHQ [18] at 256$\times$256, 500k LSUN-bedroom [24] at
256$\times$256, and 8k VGG flowers [25] at 128$\times$128\. Hence, GAN models
were trained to generate fake facial images, fake bedrooms and fake flowers.

With regard to the pixel values of the generated images, in our experiments,
they range in $[0,1]$. We denote with $S\times S$ the output dimension of the
generator, that is the size of the generated image. Experiments are carried
out considering different sizes/resolutions for the generated images, namely
different values $S$.

Regarding the payload, in our experiments, we consider a watermark length of
100 and 50 embedded bits.

As we said, in order to increase the robustness of the embedded watermark
against image manipulations, we also trained a version of the watermarked GAN
where we augment the generated images before watermark decoding, via the
processing layer (see Fig.1). The processing operations include the addition
of Gaussian noise, with standard deviation $\sim$[0.001, 0.15], Gaussian
blurring, with kernel size $\sim$[0, 9], and standard deviation in $\sim$[5,
15], JPEG compression with quality factors approximately in $\sim$[20, 50],
and color transformations including brightness, contrast and saturation
adjustment in the range [1.0, 1.30]. The processing operations above are
implemented by using Torchvision [26], and the JPEG compression is realized
using a differentiable approximation in [13]. Each processing is applied with
the probability of 15%.

###  IV-B Details and performance of the watermarking networks

As we said, every watermark decoding function can be used to supervise GAN-
watermarking. Following Yu [6], we choose StegaStamp [12] (this choice also
eases the comparison between the two GAN watermarking methods). Although
StegaStamp has good generalization capabilities, being able to encode and
decode images coming from domains different than those used for training, in
order to achieve the best performance in terms of quality of the watermarked
images and bit accuracy of the watermark, we considered a matched scenario and
we trained StegaStamp on the same dataset later used for GAN watermarking.

In all cases, StegaStamp is trained for 30 epochs with batch size 64, using
Adam optimizer and learning rate 1e-3. The performance of the trained
StegaStamp watermarked network is reported in Table II. The PSNR is always
above 45 and the SSIM above 99%, proving that the encoder is able to embed the
watermark with almost no impact on image quality. Meanwhile, the decoder can
extract the watermark with perfect accuracy.

TABLE II: Performances of the watermarking network. The image resolution is 128 $\times$ 128, the watermark length is 100. The MSE is calculated with pixel range [0,1] Dataset | MSE | Bit Acc | PSNR | SSIM(%)
---|---|---|---|---
CelebA | 1.2e-5 | 100.00 | 45.31 | 99.48
LSUN-bedroom | 5.0e-6 | 100.00 | 48.59 | 99.69
Flowers | 5.0e-6 | 100.00 | 47.60 | 99.53
Figure 2: Some examples of: (a) samples generated by original GAN, (b) samples
generated by the GAN watermarking network without augmentation, (c) samples
generated by the GAN watermarking network with augmentation.

###  IV-C Results

We evaluated the performance of our GAN watermarking scheme in terms of image
quality, namely, the invisibility of the watermark, and watermark bit
accuracy, defined as the average percentage of bits correctly recovered by the
decoder. The robustness of the embedded watermark is also assessed in Section
IV-D.

Tables III shows the performance of the proposed method applied to different
GAN architectures and datasets, i.e., generation tasks, also considering
different resolutions $S$ of the generated images. The payload is fixed to 50
and 100. The performances of the network trained both with and without the
processing layer are reported, labeled as w. and w/o aug respectively. As it
can be seen, thanks to the supervised embedding, we get nearly perfect
watermark bit accuracy (around 99%) in all the cases. The quality of the
images generated with our watermarking GAN is similar to the quality of the
images generated by the original GAN, proving that the embedded watermark is
indeed invisible. Some examples of generated images are provided in Fig. 2. We
see that the watermarked images are visually indistinguishable from those
generated by the non-watermarked model.

TABLE III: Bit accuracy of our method on different architectures and datasets. Dataset |  Model |  Resolution (S) |  Payload (bit) |  Bit Acc % -w/o aug |  Bit Acc % -with aug
---|---|---|---|---|---
CelebA |  BEGAN |  64 |  50 |  99.83 |  99.14
|  PGGAN |  128 |  50 |  99.01 |  99.20
|  StyleGAN2 |  128 |  50 |  99.74 |  99.57
LSUN bedroom |  PGGAN |  128 |  50 |  99.61 |  99.81
|  StyleGAN2 |  256 |  50 |  99.40 |  99.23
|  |  256 |  100 |  99.23 |  99.12
FFHQ |  StyleGAN2 |  128 |  50 |  99.83 |  99.40
|  |  256 |  50 |  99.46 |  99.39
|  |  256 |  100 |  99.29 |  99.74
102Flowers |  StyleGAN2 |  128 |  50 |  99.50 |  99.05
|  |  128 |  100 |  99.29 |  99.62

We compared our method with [6], whose results are obtained using the pre-
trained models and the code released by the authors. The results are reported
in Table IV in terms of bit accuracy for two GAN generation tasks/datasets,
namely CelebA and LSUN bedroom, and with two different architectures, namely
PGGAN and StyleGAN2. For all the cases reported in the table, the image size
is 128 $\times$ 128, and the payload is equal to 100 bits. We see that
performances are good for both methods, with our method achieving slightly
superior performance in some cases, e.g. with PGGAN trained on LSUN-bedroom.
The reason for the better results achieved by our method is that in our scheme
the GAN is forced to inject the watermark inside the images due to the
presence of the decoding error term. This is not the case with the method in
[6] where the GAN is trained normally, i.e., with the standard loss, on
watermarked images, without any direct control on embedding.

TABLE IV: Comparison with the state-of-the-art in terms of watermark bit accuracy.  Dataset |  Model |  Method |  Bit Acc -w/o aug |  Bit Acc -with aug
---|---|---|---|---
CelebA |  PGGAN |  Ours |  99.09 |  99.45
|  |  [6] |  98.00 |  N/A
|  StyleGAN2 |  Ours |  99.85 |  99.62
|  |  [6] |  99.00 |  N/A
LSUN bedroom |  PGGAN |  Ours |  99.29 |  99.47
|  |  [6] |  93.00 |  N/A
|  StyleGAN2 |  Ours |  99.10 |  99.22
|  |  [6] |  99.00 |  N/A
Figure 3: Performances (watermark bit accuracy) against four common image
processing applied with different strengths. The results of GAN watermarked by
the original decoder and augmented decoder is denoted by blue curve and yellow
curve. Results of Yu et al. [6] are marked in green. We use black dashes line
to plot the PSNR with values on the right Y-axis.

###  IV-D Robustness analysis

Post-processing applied to the generated images may remove the watermark and
then reduce the bit accuracy of the decoded watermark, eventually disabling
watermark verification.

The results achieved by our method for the case of PGGAN trained on CelebA
with $S=128$ and payload 100 are shown in Fig. 3 (similar performances are
obtained in the other cases), where they are also compared with the method in
[6]. The results with and without the processing layer (i.e., w and w/o
augmentation) are reported for our scheme. The black dashed lines show the
PSNR between the original images and the processed images for the various
processing parameters (strengths) reported on the x-axis.

We see that, when trained with the processing layer, the watermark embedded in
the generated images via the proposed GAN is very robust against processing,
with improved robustness with respect to [6]. In particular, the watermark bit
accuracy for our scheme remains above 75% under a relatively wide range of
perturbations, and decreases below this value when the distortion introduced
in the image is very high (low PSNR), thus making the image unusable. For
example, the bit accuracy is about 70% when the generated images are JPEG
compressed with a quality factor of 50, which corresponds to a heavy
compression in practice. Fig. 4 shows some examples of processed images and
the corresponding bit accuracy of the retrieved watermark under very strong
attacks in the various cases.

Figure 4: Examples of manipulated images under different post-processing
operations, and corresponding bit accuracies. The original image is generated
with our watermarking PGGAN trained on CelebA with the processing layer.

##  V Conclusions

In this paper, we propose a method for watermark embedding of GAN models that
can be used for the protection of the IP of GANs. The method works by training
the GAN in such a way that any image generated by the GAN contains an
invisible watermark (signature). Watermark embedding is achieved in a
supervised manner, by exploiting a pre-trained watermark decoder and using the
decoding error to guide the training of the GAN. Apart from [6], where a
similar goal has been pursued in the attempt to provide a proactive solution
for GAN detection, this is the first method focusing on the protection of the
IPR of GANs. The proposed method is general and can be used to protect the IPR
of any GAN architecture. Our experiments, carried out on several GAN
architectures and tasks, reveal that the watermark embedded in the generated
images via our method is invisible and robust against several common post-
processing operations. The watermark is embedded by fine-tuning, thus greatly
reducing the computational burden and allowing GAN owners to easily protect
already trained GANs, that is a peculiarity and a noticeable strength of the
method.

Interestingly, the proposed method can also represent a proactive solution for
GAN image detection and attribution, and the presence of the prescribed
signatures (watermarks) inside the image can be checked to attribute the image
to the generative model that produced it.

Future work will be devoted to investigating the limits in terms of watermark
capacity for the various architectures, that is, to characterize the maximum
number of bits that can be embedded in the output without impairing the visual
quality of the generated images. The use of channel coding to improve the bit
accuracy under strong attacks is also worth investigating.

## Acknowledgment

This work is supported in part by the National Key Research and Development
Plan of China under Grant 2020YFB1005600,in part by the National Natural
Science Foundation of China under grant numbers 62122032, and U1936118, in
part by Qinglan Project of Jiangsu Province, and “333” project of Jiangsu
Province. This work has been partially supported by the China Scholarship
Council (CSC), file No.202109040029.

## References

  * [1] Y. Uchida, Y. Nagai, S. Sakazawa, and S. Satoh, “Embedding watermarks into deep neural networks,” in _Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval_ , 2017, pp. 269–277.
  * [2] J. Wang, H. Wu, X. Zhang, and Y. Yao, “Watermarking in deep neural networks via error back-propagation,” _Electronic Imaging_ , vol. 2020, no. 4, pp. 22–1, 2020.
  * [3] Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in _27th USENIX Security Symposium (USENIX Security 18)_ , 2018, pp. 1615–1631.
  * [4] B. D. Rouhani, H. Chen, and F. Koushanfar, “Deepsigns: an end-to-end watermarking framework for protecting the ownership of deep neural networks,” in _ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019.
  * [5] Y. Li, B. Tondi, and M. Barni, “Spread-transform dither modulation watermarking of deep neural network,” _Journal of Information Security and Applications_ , vol. 63, p. 103004, 2021.
  * [6] N. Yu, V. Skripniuk, S. Abdelnabi, and M. Fritz, “Artificial fingerprinting for generative models: Rooting deepfake attribution in training data,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 448–14 457.
  * [7] I. J. Cox, J. Kilian, F. T. Leighton, and T. Shamoon, “Secure spread spectrum watermarking for multimedia,” _IEEE transactions on image processing_ , vol. 6, no. 12, pp. 1673–1687, 1997.
  * [8] V. Holub and J. Fridrich, “Designing steganographic distortion using directional filters,” in _2012 IEEE International workshop on information forensics and security (WIFS)_. IEEE, 2012, pp. 234–239.
  * [9] M. Barni, F. Bartolini, V. Cappellini, and A. Piva, “A dct-domain system for robust image watermarking,” _Signal processing_ , vol. 66, no. 3, pp. 357–372, 1998.
  * [10] E. Ganic and A. M. Eskicioglu, “Robust dwt-svd domain image watermarking: embedding data in all frequencies,” in _Proceedings of the 2004 Workshop on Multimedia and Security_ , 2004, pp. 166–174.
  * [11] M. Barni, F. Bartolini, and A. Piva, “Improved wavelet-based watermarking through pixel-wise masking,” _IEEE transactions on image processing_ , vol. 10, no. 5, pp. 783–791, 2001.
  * [12] M. Tancik, B. Mildenhall, and R. Ng, “Stegastamp: Invisible hyperlinks in physical photographs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2117–2126.
  * [13] J. Zhu, R. Kaplan, J. Johnson, and L. Fei-Fei, “Hidden: Hiding data with deep networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 657–672.
  * [14] M. Barni, F. Pérez-González, and B. Tondi, “Dnn watermarking: four challenges and a funeral,” in _Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security_ , 2021, pp. 189–196.
  * [15] Y. Li, H. Wang, and M. Barni, “A survey of deep neural network watermarking techniques,” _Neurocomputing_ , vol. 461, pp. 171–193, 2021.
  * [16] D. S. Ong, C. S. Chan, K. W. Ng, L. Fan, and Q. Yang, “Protecting intellectual property of generative adversarial networks from ambiguity attacks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3630–3639.
  * [17] H. Wu, G. Liu, Y. Yao, and X. Zhang, “Watermarking neural networks with watermarked images,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 7, pp. 2591–2601, 2020.
  * [18] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [19] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8110–8119.
  * [20] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” in _International Conference on Learning Representations_ , 2018.
  * [21] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [22] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [23] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * [24] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” _arXiv preprint arXiv:1506.03365_ , 2015.
  * [25] M.-E. Nilsback and A. Zisserman, “Delving deeper into the whorl of flower segmentation,” _Image and Vision Computing_ , vol. 28, no. 6, pp. 1049–1062, 2010.
  * [26] S. Marcel and Y. Rodriguez, “Torchvision the machine-vision package of torch,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1485–1488.