#  ParaGen : A Parallel Generation Toolkit

Jiangtao Feng* * Work was done at ByteDance. Shanghai AI Laboratory  Yi Zhou
ByteDance Inc.  Jun Zhang* Shanghai AI Laboratory  Xian Qian ByteDance Inc.
Liwei Wu ByteDance Inc.  Zhexi Zhang ByteDance Inc.
Yanming Liu* Shanghai Jiaotong University  Mingxuan Wang ByteDance Inc.  Lei
Li* Univeristy of California Santa Barbara  Hao Zhou* Insititute for AI
Industry Research, Tsinghua University

###### Abstract

ParaGen is a PyTorch-based NLP toolkit for further development on parallel
generation. ParaGen provides thirteen types of customizable plugins, helping
users to experiment quickly with novel ideas across model architectures,
optimization, and learning strategies. We implement various features, such as
unlimited data loading and automatic model selection, to enhance its
industrial usage. ParaGen is now deployed to support various research and
industry applications at ByteDance. ParaGen is available at
https://github.com/bytedance/ParaGen.

##  1 Introduction

Recently, neural sequence generation model achieve great success Vaswani et
al. (2017); Lewis et al. (2020); Liu et al. (2020). Among a surge of sequence
generation algorithms, parallel generation or non-autoregressive generation
methods gain increasing attention on various tasks for high inference speed Gu
et al. (2018); Saharia et al. (2020); Qian et al. (2021a); Gu and Kong (2021);
Huang et al. (2022) and competitive performance against auto-regressive
transformer Gu et al. (2019); Chan et al. (2020); Qian et al. (2021b). Apart
from natural language processing, parallel generation also demonstrates its
superiority and scalability on text-to-speech synthesis Ren et al. (2021) and
high-resolution image synthesis Chang et al. (2022).

Several toolkits on sequence generation has been presented for developing
sequence generation algorithms, such as FairSeq Ott et al. (2019),
Tensor2Tensor Vaswani et al. (2018), Transformers Wolf et al. (2020) and
OpenNMT Klein et al. (2017). These toolkits are mostly born with auto-
regressive transformers with maximum likelihood estimation training and are
used for research purposes.

Figure 1: Overall framework of ParaGen . The data retrieval block is colored
with green; the neural network modules are colored with yellow; the process
scheduling processing block is colored with bleu; and the task, which
dominates all the processes, is colored with red. All the white blocks here
are customizable classes in ParaGen .

In this paper, we present ParaGen , an extensible toolkit for parallel
generation, which is first developed with Glancing Transformer on WMT-21
Competition Qian et al. (2021b). We redesign the code architecture for easy
modification on training and decoding methods, such as glancing training Qian
et al. (2021a), imitation learning Wei et al. (2019), inference algorithms
(noisy parallel decoding) Gu et al. (2018), and mask-predict decoding
Ghazvininejad et al. (2019), which are critical to enhancing parallel
generation algorithm developments. Besides, ParaGen also suits industrial
usage with robust implementations and attractive features, such as unlimited
data loading, asynchronized input/output, plug-in Huggingface
tokenizers/models Wolf et al. (2020) and fast training/inference with LightSeq
Wang et al. (2021, 2022). Apart from parallel generation, ParaGen also
reproduces typical tasks with step-by-step scripts, such as autoregressive
translation Vaswani et al. (2017), text summarization Lewis et al. (2020),
text classification Wang et al. (2018), and extractive question answering
Rajpurkar et al. (2016). As for large-scale pretraining, ParaGen supports BERT
pretraining Devlin et al. (2019), and multilingual translation with mBART
pretraining Liu et al. (2020). ParaGen is now deployed to support various
research and industrial applications at ByteDance.

##  2 Architecture Design

The overall architecture of ParaGen is shown as Figure 1. ParaGen consists of
four main functional blocks: data, model, trainer, and evaluator. The data
block focuses on data input, processing, sampling, and loading; the model
block consists of neural models in training and inference; the trainer is
implemented for scheduling the training process; the evaluator defines the
evaluation metrics. Compared with the previous frameworks, we offer 13 types
of plug-ins across the three blocks, which makes ParaGen more extensible for
experimenting with new ideas.

###  2.1 Data

We design the data organization block on four base concepts, including
reading, preprocessing, sampling strategy and loading, deriving four
customizable class or functions respectively, i.e. Dataset, Data Processing,
Sampler and DataLoader. We address ParaGen ’s data process paradigm along with
two key topics: online-offline data processing and unlimited data loading
challenge.

#### Dataset

The Dataset instances read data and organize it to a dict-format object,
despite their storage format on disks. Users are allowed to develop their on
Dataset class for customization usage by implementing load and callback
functions. Currently, ParaGen supports data stored in various formats,
including raw texts, parallel texts, and JSON files. The Datasets as well as
other classes in ParaGen co-work with an underlying io module to suit
different file systems, reading and writing data on a local disk or a Hadoop
file system. It is worth noting that the io module is also modularized and
extensible to suit data input/output under more scenarios. Besides, we also
develop StreamingDataset, reading data in a streaming way. The
StreamingDataset can read extremely large-scale data with constant memory
consumption, making it extensible to industrial usage.

#### Data Processing

Data preprocessing, such Byte-Pair Encoding Sennrich et al. (2016), is
critical to sequence generation and varies from task to task. To enhance task-
specific data preprocessing, ParaGen provides interfaces within Task class to
allow customization. The data processing is roughly divided into two
categories, offline data processing as data_collate_fn and online data
processing collate_fn. The data_collate_fn refers to offline data processing
and proceeds before the training/inference stage start with input from
Dataset. Thus data processed by data_collate_fn remains unchanged during the
training/inference process, which speeds up training and inference by
eliminating repeated data processing. The collate_fn is designed as online
processing to enhance flexibility and to allow users to adjust data processing
strategies, such as batching, during training and inference. We believe the
combination of offline and online data processing would make data processing
more flexible and extensible.

#### Sampler

The sampling strategy is a non-negligible algorithm in the online data
processing. Although PyTorch provides a base class of sampling strategy, it is
still often ignored by existing generation frameworks. ParaGen allows users to
develop their sampling strategies by implementing a Sampler instance to decide
how data are organized into batches. A technical challenge of incorporating
customizable sampling strategies is their compatibility with the feature of
unlimited data loading. We solve this problem in the DataLoader with a cache
mechanism.

#### DataLoader

DataLoader is the final stage of data processing and the beginning of neural
model processing, acting as a bridge to connect data and neural models. It can
also be viewed as a coordinator of data processing. It first fetches a batch
of samples, according to the sampling strategy determined by Sampler, from
data memory with offline processed data. Then it sends the data batches to
online data processing, which becomes a private object of DataLoader instance
at initialization, and gets a batch to feed the neural network. However, in
the original PyTorch, DataLoader is incompatible with streaming data loading.
We extend the dataloader and implement a StreamingDataLoader to read data
streamingly, further featuring unlimited data loading.

###  2.2 Module

A key principle of designing ParaGen ’s model block is based on the concept
separation of training and inference. Unlike classification models, sequence
generation ones become different when they are applied in training and
inference. For example, a sequence generation model is trained in a teacher-
forcing way whilst it generates sequences with the help of the beam search
algorithm. Thus we defines four sub-modules for ParaGen model block: Model,
Generator, Criterion and Search. Model and Criterion are often used in
training process whereas Generator and Search are in inference. We argue that
the training-inference separation of neural models would benefit the
downstream neural model optimization Wang et al. (2021) in industrial usage
without harming flexibility in developing new models.

#### Model

We implement the architecture of neural models with learnable parameters by
inheriting ParaGen’s Model class. Similar to the models in the existing, it
consumes a batch of samples and produces logits over the predicted target. We
provide several popular implementations, including parallel generation,
autoregressive sequence generation, extraction model, and sequence
classification/regression. Besides, Huggingface models at present are widely-
used for their large-scale pre-training models, and we implemented a
Huggingface model wrapper to make it compatible in ParaGen .

#### Generator

In ParaGen , we advocate Generator, instead of the original Model, to apply to
the inference stage. Generator is designed as a wrapper to Model with extra
decoding algorithms, such as beam search and greedy search in autoregressive
sequence generation Vaswani et al. (2017), noisy padding mask in parallel
generation Gu et al. (2018), and even extraction algorithms in extractive
tasks. Although the extra decoding algorithms could also be implemented as
post-processing, they would benefit from tensor computation on GPUs, which
further speeds up the computations. It is also recommended in ParaGen to have
decoding algorithms on GPUs and post-processing on CPUs work with each other
to achieve the tradeoff between flexibility and speedup. We here argue that it
is essential for industrial usage to separate Generator from Model, because
Generator, which is gradient-free, requires more elaborated and extreme
optimization to enhance efficiency. Previous study Wang et al. (2021) shows
that joint optimization on neural models and decoding algorithms achieves
significant speedup and reduces GPU memory consumption. Nevertheless, such
joint optimization does not suit Model in training. In ParaGen , we implement
Generators for sequence extraction, auto regressive sequence generation and
parallel sequence generation.

#### Criterion

Like previous frameworks, we define Criterion class as the objective functions
across various tasks. It measures the divergence between predicted logits and
golden reference. From the Criterion, we compute the gradients of all the
learnable parameters for optimization. We also allow neural models, such as
Huggingface models, to compute loss by themselves. Due to the modularization
in ParaGen , Criterions can be combined to enhance multi-task learning.

#### Search

Search or decoding algorithms, such as beam search and noise padding mask, are
critical to sequence generation. We modularize Search to support users in
developing their awesome sequence search algorithms, for both autoregressive
or parallel generation, more than existing ones. The Search algorithms act as
a part of Generator, co-working with Model to produce final sequences.

###  2.3 Trainer

One big difference between ParaGen and existing learning framework is
customizing the training process. Recent research shows that a neural model
with the same architecture trained with a well-designed training strategy
performs significantly better. Customizing a Trainer helps the users to
experiment with training strategies conveniently. The Trainer formulates the
whole training process and includes several types of customization: loss
computation, optimizer, and rate scheduler.

#### Loss computation

Trainer leaves an interface forward_loss for implementation of loss
computation. Elaboration on loss computation is critical to deep learning
algorithms to enhance models’ performance. For examples, a) GLAT Qian et al.
(2021a) can be used for computing a three-stage objective, glancing at the
target, modifying neural network inputs/targets, and learning; b) FreeLB Zhu
et al. (2020) adopts an adversarial gradient to inject to neural model to
learn it robustly; c) CoNT An et al. (2022) leverage a generation process to
adopt contrastive learning to enhance sequence generation. Thus we believe
that customization on loss computation frees developers from the stereotyped
training process and encourage new experimental training algorithms.

#### Optimizer

ParaGen provides Optimizer customization following the original PyTorch. All
the optimizers implemented in PyTorch and Huggingface can be used directly,
and experimental optimizers are also encouraged. The coordination of Optimizer
to advanced optimization algorithms, such as mix-precision training, apex
support, and distributed training, is automatic.

#### Rate Scheduler

We implement a functional tool called Rate Scheduler to define how a rate
typed as an integer or float is scheduled. The rate in ParaGen could be any
hyper-parameter beyond the learning rate, allowing users to schedule their
training process more flexibly. In design, the original PyTorch treats the
optimizer as an attribute to the learning rate scheduler, but ParaGen does
this differently by setting the scheduled learning rate as a part of the
optimizer. The learning rate is actively scheduled through the interaction
between Optimizer and Trainer.

###  2.4 Evaluator

Evaluator formulates the overall evaluation process in ParaGen , supporting
customization on two aspects, including data sets and metrics. In evaluations,
we compute the Cartesian product of data sets and metrics to obtain several
performance scores. The scores are averaged to obtain an overall judgment on
the current neural model. These scores will further return to Trainer for
model selection.

#### Metric

We provide Metric for independent evaluation of the divergence between
predicted hypotheses and ground-truth references. Metric in ParaGen can be
designed as lexical metrics (BLEU Papineni et al. (2002a) and Rouge Lin
(2004)), numeric metrics (Accuracy and F1-score), and model-based metrics
(BERT-score Zhang et al. (2019))

###  2.5 Difference to FairSeq

A close sequence generation toolkit to ParaGen is FairSeq Ott et al. (2019).
ParaGen differs from FairSeq in four aspects: a) FairSeq is for general
purpose sequence generation, while ParaGen is carefully designed for parallel
generation; b) Fairseq’s standard pipeline requires binarizing the data first,
which is extremely fast and efficient for training. Such a way may make on-
the-fly data manipulation a bit difficult. ParaGen allows reading the raw data
and modifying it dynamically in the training loop, without sacrificing the
speed. c) FairSeq supports 5 types of customizable modules whereas ParaGen
offers 13 classes. Fairseq provides a unified training loop, ParaGen
disentangles the training and inference process into independent modules
(Trainer, Evaluator), so that the customization for each task is more user-
friendly. d) ParaGen implements more features specially designed for
industrial usage.

##  3 Implementation & Features

ParaGen implements fundamental functions, such as distributed training on
multiple machines and GPUs (with Horovod Sergeev and Balso (2018)), mix-
precision training (with apex111https://github.com/NVIDIA/apex), incremental
decoding for the autoregressive models, breakpoint resuming, out-of-memory
recovery, early stopping, and accumulated gradients. Moreover, we also
implement advanced functions, including unlimited data loading, automatic
model selection, multi-task training, and fast training/inference with
LightSeq. In this section, we focus on these advanced features.

#### Unlimited Data Loading

Pretrained models has show its strong capability in generalizing to new tasks
Devlin et al. (2019); Liu et al. (2019); Radford et al. (2019); Lewis et al.
(2020); Raffel et al. (2020); Floridi and Chiriatti (2020). However, such pre-
trained models usually demand billions, trillions, and even more of the
training data. Moreover, for industrial usage, the large amount of internal
data is a challenge for data loading. In ParaGen , we implement a
StreamingDataLoader that reads data from disk in a streaming way with the file
input stream. It also features data distribution with multi-GPU training and
local shuffling for data batching. With the help of StreamingDataLoader,
ParaGen can read unlimited data with limited memory usage.

#### Automatic Model Selection

ParaGen automatically selects the best models with a customized assessment
metric during the training process. By providing the trainer with an assess_by
arguments as {DATA_NAME}.{METRIC_NAME}, the trainer picks up the checkpoints
that performs the best on {DATA_NAME} with respect to {METRIC_NAME}. Note that
once a checkpoint is selected to save, ParaGen also saves the average of $k$
checkpoints before and marks the averaged checkpoints as best_avg222We find it
performs better compared with the average of the last-$k$ checkpoints after
the training process ends..

#### Asynchronized Input/Output

ParaGen uses an asynchronized input/output implementation for reading and
writing to maximize the utility of GPU resources. Asychronized output is
important for model selection. Because the computation of best_avg checkpoints
costs a long time to finish, especially for large models in industrial
applications.

#### Multi-Task Learning

ParaGen provides an easy-to-use way for multi-task learning. ParaGen
implements a MultiTaskCriterion which automatically combines a list of
criteria.

#### Plug-in Hugginface Tokenizers and Models

Huggingface Wolf et al. (2020) is a widely used pre-trained model library and
demonstrates its effectiveness among various tasks. In ParaGen , Hugginface
can be directly used in an import-register way. It allows researchers to
develop algorithms upon pre-trained models and to use advanced features
provided in ParaGen .

#### Fast Training and Inference with LightSeq

In ParaGen , researchers and developers can use LigthSeq easier to speedup
their transformer models/modules in training Wang et al. (2022) and inference
Wang et al. (2021). Without understanding the details of LightSeq, users can
speed up their transformer models/modules by simply appending a LS prefix to
the model/module class name in the configuration.

##  4 Reproducibility

ParaGen can be used for various tasks beyond parallel generation. We provide
reproducible results and scripts on six benchmarks on ParaGen , including: a)
glancing transformer on WMT14 En-De; b) transformer on IWSLT14 De-En, WMT14
En-De, and WMT14 En-Fr; b) transformer on Multi-News and XSum; c) mBART on
Multilingual Translation; d) plug-in Huggingface on SQuAD 1.1; e) BERT
pretraining and fine-tuning on GLUE benchmark. For simplicity, we eliminate
the details of reproduction configurations and hyper-parameters. The
reproduction results are shown in Appendix A and scripts can be found in our
repository.

##  5 Conclusions

We present the ParaGen toolkit for parallel sequence generation. It supports
13 types of modules for customization and advocates a plug-in usage for
further development. Its robust implementation and features enhance the
research algorithm design and industrial development. In the future, we will
create more plugins to extend ParaGen to more research areas.

## References

  * An et al. (2022) Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. 2022.  Cont: Contrastive neural text generation.  _arXiv preprint arXiv:2205.14690_.
  * Chan et al. (2020) William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. 2020.  Imputer: Sequence modelling via imputation and dynamic programming.  In _International Conference on Machine Learning_ , pages 1403–1413. PMLR.
  * Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. 2022.  Maskgit: Masked generative image transformer.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  Gpt-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ , 30(4):681–694.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019.  Mask-predict: Parallel decoding of conditional masked language models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6112–6121, Hong Kong, China. Association for Computational Linguistics.
  * Gu et al. (2018) Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018\.  Non-autoregressive neural machine translation.  In _International Conference on Learning Representations_.
  * Gu and Kong (2021) Jiatao Gu and Xiang Kong. 2021.  Fully non-autoregressive neural machine translation: Tricks of the trade.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 120–133, Online. Association for Computational Linguistics.
  * Gu et al. (2019) Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019.  Levenshtein transformer.  _Advances in Neural Information Processing Systems_ , 32.
  * Huang et al. (2022) Chenyang Huang, Hao Zhou, Osmar Zaiane, Lili Mou, and Lei Li. 2022.  Non-autoregressive translation with layer-wise prediction and deep supervision.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002a) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL_.
  * Papineni et al. (2002b) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Qian et al. (2021a) Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021a.  Glancing transformer for non-autoregressive neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1993–2003, Online. Association for Computational Linguistics.
  * Qian et al. (2021b) Lihua Qian, Yi Zhou, Zaixiang Zheng, Yaoming Zhu, Zehui Lin, Jiangtao Feng, Shanbo Cheng, Lei Li, Mingxuan Wang, and Hao Zhou. 2021b.  The volctrans GLAT system: Non-autoregressive translation meets WMT21.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 187–196, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ren et al. (2021) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021\.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  In _International Conference on Learning Representations_.
  * Saharia et al. (2020) Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020.  Non-autoregressive machine translation with latent alignments.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1098–1108, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Sergeev and Balso (2018) Alexander Sergeev and Mike Del Balso. 2018.  Horovod: fast and easy distributed deep learning in TensorFlow.  _arXiv preprint arXiv:1802.05799_.
  * Vaswani et al. (2018) Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018.  Tensor2Tensor for neural machine translation.  In _Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 193–199, Boston, MA. Association for Machine Translation in the Americas.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2022) Xiaohui Wang, Yang Wei, Ying Xiong, Guyue Huang, Xian Qian, Yufei Ding, Mingxuan Wang, and Lei Li. 2022.  Lightseq2: Accelerated training for transformer-based models on gpus.  In _Proceedings of The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’22)_.
  * Wang et al. (2021) Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. 2021.  LightSeq: A high performance inference library for transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_ , pages 113–120, Online. Association for Computational Linguistics.
  * Wei et al. (2019) Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019.  Imitation learning for non-autoregressive neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1304–1312, Florence, Italy. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations, ICLR_.
  * Zhu et al. (2020) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020.  Freelb: Enhanced adversarial training for natural language understanding.  In _International Conference on Learning Representations_.

##  Appendix A Reproducible Results

We implement typical models on various tasks: a) glancing transformer on WMT14
En-De; b) transformer on IWSLT14 De-En, WMT14 En-De, and WMT14 En-Fr; b)
transformer on Multi-News and XSum; c) mBART on Multilingual Translation; d)
plug-in Huggingface on SQuAD 1.1; e) BERT pretraining and fine-tuning on GLUE
benchmark. These models are implemented with reproducible scripts. The
detailed reproducible results are shown in the rest of the section.

###  A.1 Glancing Transformer on WMT14 En-De

We trained Glancing Transformer on Transformer-AT distilled data. We report
sacrebleu Post (2018) and tokenized BLEU Papineni et al. (2002b) for
completeness.

Model | sacrebleu | tok bleu
---|---|---
GLAT | 24.40 | 24.98
GLAT + avg-ckpt | 24.58 | 25.29
Table 1: BLEUs on distilled WMT14 En-De. GLAT+avg-ckpt is the average checkpoint among the last 10 checkpoints. Task | Model | sacrebleu | tok bleu
---|---|---|---
IWSLT14 De-En | small | 33.1 | 34.5
WMT14 En-De | base | 26.9 | 27.5
WMT14 En-De | big | 27.7 | 28.4
WMT14 En-Fr | big | 40.3 | 43.3
Table 2: BLEUs on machine translation benchmarks.

###  A.2 Transformer on Machine Translation

We implement widely-used transformer architecture and provide its results on
IWSLT/WMT benchmarks. For IWSLT14 De-En, we use transformer-small
architecture; for WMT14 En-De and WMT14 En-Fr, we use transformer-big
architecture. Similar to Glancing Transformer, we report its results with
sacrebleu and tokenized bleu.

###  A.3 mBART on Multilingual Translation

ParaGen provides implementation to pre-train mBART from scratch. We finetune
the pre-trained mBART on XX-en translation tasks.

Language Pairs | mBART | mBART + avg-ckpt
---|---|---
de-en | 41.45 | 41.84
fr-en | 39.09 | 39.41
ja-en | 21.68 | 22.84
pl-en | 32.03 | 32.50
ro-en | 36.75 | 37.24
mn-en | 12.75 | 14.14
hi-en | 26.34 | 27.78
Table 3: Results on XX-en translation with mBART pre-trained from scratch.

###  A.4 Abstractive Text Summarization

task | model | rouge-1 | rouge-2 | rouge-n
---|---|---|---|---
Multi-News | tr-base | 33.59 | 5.91 | 30.71
BART-base | 46.80 | 17.93 | 43.01
XSum | bart-base | 42.49 | 19.52 | 34.37
Table 4: Results on abstractive text summarization.

###  A.5 Question Answering

ParaGen also provides results on extractive question answering with plug-in
Huggingface models. We test various pre-trained models in Huggingface on SQuAD
1.1.

Model | F1 | Exact Match
---|---|---
BERT-base-uncased | 88.31 | 81.02
BERT-base-cased | 88.34 | 81.11
BERT-large-cased | 90.73 | 83.87
RoBERTa-base | 91.88 | 85.34
RoBERTa-large | 93.44 | 87.26
BART-base | 91.27 | 84.52
BART-large | 93.14 | 86.70
Table 5: Plug-in Huggingface Pretrained Models on SQuAD 1.1

###  A.6 BERT

ParaGen supports training BERT from scratch with customized data. We train
BERT on two data sets, news and wibo, and test it on GLUE benchmarks Wang et
al. (2018). We use standard evaluation metrics for each task.

Task | Metric | news | wibo
---|---|---|---
CoLA | Matthews’ Corr | 61.82 | 58.36
SST-2 | Accuracy | 93.00 | 91.86
STS-B | Pearson/Spearman Corr | 86.97/86.88 | 87.95/87.65
MRPC | F1/Accuracy | 91.13/87.75 | 91.13/87.50
QQP | F1/Accuracy | 87.95/91.05 | 88.05/91.16
MNLI-m/mm | Accuracy | 83.42/83.28 | 83.24/83.06
QNLI | Accuracy | 89.22 | 90.81
RTE | Accuracy | 67.15 | 61.37
Table 6: Results on GLUE by BERT pre-trained on news and wibo dataset.


# Hide and Seek: How Does Watermarking Impact Face Recognition?

Yuguang Yao∗ Steven Grosz∗ Sijia Liu Anil Jain
Michigan State University

###### Abstract

The recent progress in generative models has revolutionized the synthesis of
highly realistic images, including face images. This technological development
has undoubtedly helped face recognition, such as training data augmentation
for higher recognition accuracy and data privacy. However, it has also
introduced novel challenges concerning the responsible use and proper
attribution of computer generated images. We investigate the impact of digital
watermarking, a technique for embedding ownership signatures into images, on
the effectiveness of face recognition models. We propose a comprehensive
pipeline that integrates face image generation, watermarking, and face
recognition to systematically examine this question. The proposed watermarking
scheme, based on an encoder-decoder architecture, successfully embeds and
recovers signatures from both real and synthetic face images while preserving
their visual fidelity. Through extensive experiments, we unveil that while
watermarking enables robust image attribution, it results in a slight decline
in face recognition accuracy, particularly evident for face images with
challenging poses and expressions. Additionally, we find that directly
training face recognition models on watermarked images offers only a limited
alleviation of this performance decline. Our findings underscore the intricate
trade off between watermarking and face recognition accuracy. This work
represents a pivotal step towards the responsible utilization of generative
models in face recognition and serves to initiate discussions regarding the
broader implications of watermarking in biometrics.

**footnotetext: Equal contribution

##  1 Introduction

Recent advancements in generative modeling, particularly in diffusion models
(DMs) [1, 2, 3], have demonstrated remarkable capabilities in synthesizing
complex and highly realistic images. However, as this technology evolves, it
brings forth new considerations regarding its safety and ethical use. These
emerging issues include harmful content generation [4, 5], deep fakes [6, 7],
privacy leakage [8, 9], copyright usurpation [10, 11, 12], and societal biases
and stereotyping [13, 14]. Therefore, significant efforts have been made to
avoid the misuse of generative models and to enhance their safety, security,
and trustworthiness [15]. For example, the watermarking technology [16, 17,
18, 19], which we will focus on in this work, entails embedding a distinctive
signature into image generation. This approach enables differentiation between
real and synthesized images and can be used to protect image copyrights and
ascertain ownership.

Figure 1:  The schematic overview of the studied face recognition pipeline,
which consists of face image generation (top), digital watermarking (middle),
and face recognition (bottom). The watermarking technique follows HiDDeN [17].

Among the myriad applications of generative models, face image generation has
garnered considerable attention recently. For example, the generation of high-
fidelity synthetic face images enables model developers to train face
recognition systems solely using synthetic data, thereby circumventing the
risk of compromising the privacy of real individuals [20]. Augmenting existing
training sets with synthesized face images can also help mitigate racial
biases and tackle data imbalance problems [21, 22]. Furthermore, generative
models have been employed to manipulate or enhance face images [23, 24, 25,
26]. Recent advances in text to image generation using denoising diffusion
probabilistic models (DDPM) have allowed for synthesizing highly realistic,
novel views and appearances of a given face image, while preserving the
identity.

Despite the extensive use of generative models in face recognition, there has
been limited research delving into the implications of integrating
watermarking into these models and how their watermarked image generations
influence their efficacy in face recognition tasks. In particular, we ask:

(Q) How effective is face recognition when evaluated on watermarked face
images?

To address (Q), we explore the influence of existing digital watermarking
technology in face recognition. The proposed system, as shown in Fig. 1,
comprises a diffusion model (DM) for face image generation, an encoder-
decoder-based network for embedding a watermark signature into synthetic
images, and a face recognition model trained on watermarked data. Assessing
through face verification tasks, aimed at determining whether two face images
belong to the same person, we observe that watermarking retains effectiveness
throughout the generation and recognition processes. Nonetheless, we do notice
a minor decline in face recognition accuracy, particularly noticeable when
comparing challenging face images.

#### Contributions.

We summarize our contributions below.

$\bullet$ We make a novel exploration into the impact of watermarked face
images on face recognition accuracy.

$\bullet$ We investigate face recognition through a unified framework that
integrates watermarking in face image generation. This facilitates us to train
and evaluate face recognition systems with watermarked synthetic data.

$\bullet$ Through extensive experiments, we showcase the effectiveness of
watermarking in face generation and recognition. We highlight its resilience
against post-hoc image transformations and offer valuable insights into the
successes and challenges encountered when employing watermarked images for
face recognition training.

##  2 Related Work

#### Face image generation.

Recent advancements in generative modeling have revolutionized the synthesis
of high-quality face images. Generative Adversarial Networks (GANs) [27] have
been extensively employed for face image generation, exemplified by notable
architectures like StyleGAN [28] and StarGAN [29]. These models adeptly
translate random noise vectors into realistic face images, facilitating the
creation of diverse and visually appealing facial representations. However,
GANs often encounter issues with training instability and mode collapse,
hindering their capacity to capture the full spectrum of facial variations.

More recently, diffusion models [30, 31] have emerged as a compelling
alternative for high-fidelity image generation. Operating by iteratively
denoising a Gaussian noise signal and guided by a learned noise schedule,
diffusion models excel in generating highly realistic and varied face images,
surpassing GANs in many respects. Prominent examples of diffusion-based face
generation models include DDPM [31] and DCFace [20]. The advent of
sophisticated face image generation techniques has introduced new avenues for
data augmentation, privacy preservation, and creative applications [32, 33,
34]. However, it has also raised concerns regarding potential misuse, such as
the creation of deepfakes [35] or identity theft. These apprehensions
underscore the necessity for methods ensuring the responsible use and
attribution of computer generated face images.

#### Digital watermarking.

Digital watermarking is a technique for embedding hidden information, known as
a watermark, into digital content. While the watermark can be designed to be
visible or invisible, we deal with watermarks which are imperceptible to human
observers but can be detected and extracted by specialized algorithms. Digital
watermarking has a wide range of applications, including copyright protection,
content authentication, and data provenance [36, 37, 38, 39]. Early works on
digital watermarking focused on developing techniques that relied on
transform-domain techniques, such as discrete cosine transform (DCT) [40, 41],
discrete wavelet transform (DWT) [42], or singular value decomposition (SVD)
[43], to embed the watermark in the frequency or singular value domain of the
image.

With the advent of deep learning, researchers have explored the use of neural
networks for digital watermarking. These learning-based approaches can
automatically learn to embed and extract watermarks from images, potentially
offering improved robustness and capacity compared to traditional methods.
Notable examples include HiDDeN [17], which uses an encoder-decoder
architecture to hide watermarks in images, and DeepStego [44], which employs
adversarial learning to enhance the imperceptibility of the watermark. There
have also been works studying watermarking for GAN images [45, 46, 47, 48,
49]. However, the impact of watermarking on face recognition has not been well
studied. Our work aims to address this gap by investigating the interplay
between digital watermarking and face recognition.

#### Face recognition.

Face recognition technology has garnered substantial attention in computer
vision research due to its diverse applications in security, surveillance, and
human-computer interaction. Early face recognition methods predominantly
relied on handcrafted features and classic machine learning algorithms.
Techniques such as Eigenfaces [50], Local Binary Patterns [51], and Histogram
of Oriented Gradients [52] have been widely used for feature extraction,
whereas classifiers like Support Vector Machines [53] and k-Nearest Neighbors
[54] were used for reocgnition.

The advent of deep learning revolutionized face recognition, where
Convolutional Neural Networks (CNNs), such as DeepFace by Taigman et al. [55],
enabled significant improvements in accuracy and robustness. Since then, many
different types of loss functions (e.g., SphereFace [56], CosFace [56],
ArcFace [57], etc.) and model architectures (e.g., VGG [58], ResNet [59], ViT
[60], etc.) for large-scale face recognition were proposed, further pushing
the boundaries of face recognition. These algorithms relied on the
availability of large-scale face recognition datasets for their success. Yet,
many of these datasets, such as the popular MS-Celeb-1M [61], have now been
recalled due to privacy concerns. This has prompted many researchers to turn
toward developing methods for realistic face image generation to replace these
large-scale face recognition datasets.

##  3 Methodology

To tackle the question (Q) posed in Sec. 1, this section introduces the
watermarking, face generation, and face recognition techniques utilized in
constructing the proposed system, as depicted in Fig. 1.

###  3.1 Digital Watermarking

Digital watermarking embeds hidden information, known as a ‘watermark’ or
‘signature’, into digital content like images. Such watermarks are often
imperceptible to humans but detectable by specialized algorithms. In the case
of (synthesized) face images, watermarking serves to track their origin and
deter unauthorized usage or distribution [16, 17, 18, 19].

To be concrete, the watermarking problem can be defined as follows: Given an
input image $\mathbf{I}\in\mathbb{R}^{H\times W\times C}$ and an $L$-bit
watermark message $\mathbf{m}\in\\{0,1\\}^{L}$ (i.e., digital signature), our
goal is to produce a watermarked image $\mathbf{I}_{\mathrm{w}}$ that
maintains visual similarity to the original image $\mathbf{I}$ while
incorporating the watermark message $\mathbf{m}$. In addition, the watermarked
image $\mathbf{I}_{\mathrm{w}}$ should be designed such that it allows for the
reverse engineering of its encoded message $\mathbf{m}$, enabling the
deciphering of the image’s origin.

Figure 2:  The schematic overview of watermarking system.

To implement the watermarking system, we employ the neural network-based
HiDDeN framework [17] due to its proven performance while any other
watermarking scheme can be used as well; see Fig. 2 for a schematic overview.
This watermarking system consists of an encoder network
$f_{\boldsymbol{\theta}}$ and a decoder network $g_{\boldsymbol{\phi}}$. The
encoder takes the input image $\mathbf{I}$ and the watermark message
$\mathbf{m}$ as inputs and generates the watermarked image
$\mathbf{I}_{\mathrm{w}}=f_{\boldsymbol{\theta}}(\mathbf{I},\mathbf{m})$. The
decoder takes the watermarked image $\mathbf{I}_{\mathrm{w}}$ as input and
reconstructs the embedded watermark message
$\hat{\mathbf{m}}=g_{\boldsymbol{\phi}}(\mathbf{I}_{\mathrm{w}})$.

The encoder and decoder networks are jointly trained using a combination of
image reconstruction loss and message decoding loss. The image reconstruction
loss $\ell_{\mathrm{recons}}$ (e.g., mean squared error) ensures that the
watermarked image is visually similar to the original image, while the message
decoding loss $\ell_{\mathrm{decode}}$ (e.g., bit-wise binary cross-entropy
loss) minimizes the difference between the embedded and extracted watermark
messages. The overall training objective of the watermarking system is then
given by

$\displaystyle\displaystyle\min_{\boldsymbol{\theta},\boldsymbol{\phi}}\,\,\mathbb{E}_{\mathbf{I},\mathbf{m}}\left[\ell_{\mathrm{recons}}(\mathbf{I}_{\mathrm{w}},\mathbf{I})+\lambda\ell_{\mathrm{decode}}(g_{\boldsymbol{\phi}}(\mathbf{I}_{\mathrm{w}}),\mathbf{m})\right],$
(1)

where recall that
$\mathbf{I}_{\mathrm{w}}=f_{\boldsymbol{\theta}}(\mathbf{I},\mathbf{m})$, and
$\lambda$ is a regularization parameter to strike a balance between the image
reconstruction and the message decoding losses. The training dataset is given
by MS-COCO [62].

It is also worth noting that during training, a random message generator is
employed to produce random bits in $\mathbf{m}$ (as depicted in Fig. 2), which
are then combined with the image input at the input of the encoder network.
The resilience to random bit messages allows us to utilize a pre-trained
encoder-decoder-based watermarking system directly to embed a user-defined
watermark message in face images, which we will employ subsequently. In
addition, the watermark training integrates data augmentations. That is,
watermarked images undergo diverse transformations, including random cropping,
resizing, and compression, mimicking real-world scenarios. Through training
with augmented data, the encoder and decoder networks learn to embed and
extract watermarks resilient to these transformations. As will be evident
later, this renders it an effective approach for watermarking generated face
images, ensuring the robustness of decoding the watermark against external
image transformations.

###  3.2 Integration of Watermark into Face Images

We begin by incorporating a designated watermark into face images. This
integration is accomplished through the encoder network
($f_{\boldsymbol{\theta}}$) of the trained watermarking system. As shown in
Fig. 3, we select the ‘$S$’ letter (represented as a binary image) as the
watermark message, with its pixel values utilized for the binary message,
denoted as $\mathbf{m}_{S}$. Therefore, given a face image $\mathbf{I}$, its
watermarked version is now given by
$\mathbf{I}_{\mathrm{w}}=f_{\boldsymbol{\theta}}(\mathbf{I},\mathbf{m}_{S})$.

Figure 3:  The embedded watermarking signature.

In addition to the specificity of the watermark message, we also consider two
sources of face images, including both synthetic and real face images. The
synthetic image source is DCFace [20], a state-of-the-art diffusion-based face
generation model. DCFace is built upon DMs (diffusion models) to enhance the
quality and controllability of generated face images conditioned on individual
identities and face variations. It excels at producing face images of a
specific identity across various styles while allowing for precise control,
thereby enriching face recognition models with augmented training data of
large intra-class variance. We utilize DCFace to simulate the scenario where
synthesized face images require integration with the watermark
$\mathbf{m}_{S}$ before their downstream applications, like face recognition.
The second image source consists of commonly-used real face image datasets,
such as CASIA-WebFace [21] and MS-Celeb-1M v2 (MS1Mv2) [57], which include
face images of real identities. We will apply watermarking to both image
sources respectively and investigate its effects on face recognition.

The performance of image watermarking is evaluated by assessing the
reconstruction accuracy of the watermark message using the decoder network
$g_{\boldsymbol{\phi}}$ applied to the watermarked image
$\mathbf{I}_{\mathrm{w}}$. In particular, the watermark is expected to exhibit
robustness against post-hoc image transformations, ensuring the reliable
extraction of the watermark message. Fig. 4 shows the effectiveness of image
watermarking against a sequence of post-watermarking data transformations
under both the synthetic image source DCFace and the real image source CASIA-
WebFace.

Figure 4:  Visualizations of watermarked images and their decoded watermark
message (the ‘$S$’ letter) based on the real image source CASIA-WebFace and
the synthetic image source DCFace. Each column represents a data
transformation operation applied after image watermarking. ‘Original’ denotes
no data transformation. ‘Crop 80%’ refers to randomly cropping the image to
80% of its original size. ‘Resize 80%’ refers to resizing the image to 80% of
its original size. ‘Bright 3x’ refers to increasing the image brightness by a
factor of 3. ‘Contrast 3x’ refers to increasing the image contrast by a factor
of 3. ‘JPEG 90%’ refers to applying JPEG compression with a quality factor of
90%.

###  3.3 Face Recognition on Watermarked Images

With a dataset of watermarked face images in hand, we proceed to conduct the
face recognition task. In this study, our face recognition setup adheres to
AdaFace [63], while any other face recognition algorithm can be used as well.
Given an input face image $\mathbf{I}$, a face recognition model
$h_{\boldsymbol{\psi}}$ maps the image to a compact feature representation
$\mathbf{z}$: $\mathbf{z}=h_{\boldsymbol{\psi}}(\mathbf{I})$, where
$\boldsymbol{\psi}$ represents the learnable parameters of the model. The
feature representation $\mathbf{z}$ is typically obtained from the penultimate
layer of a CNN, such as ResNet [59] or MobileFaceNet [64]. The model is
trained to minimize a classification loss, such as the softmax loss [57] or
the margin-based loss [56, 65, 57], which reduces the intra-class variance in
the embedding space while increasing the inter-class variance.

During inference, the learned face recognition model extracts the feature
representation $\mathbf{z}_{\mathrm{p}}$ for a probe face image
$\mathbf{I}_{\mathrm{p}}$. The similarity between the probe face
$\mathbf{I}_{\mathrm{p}}$ and a reference face $\mathbf{I}_{\mathrm{r}}$ (with
feature representation $\mathbf{z}_{\mathrm{r}}$) is computed using a
similarity metric, such as the cosine similarity

$\displaystyle
s(\mathbf{z}_{\mathrm{p}},\mathbf{z}_{\mathrm{r}})=\frac{\mathbf{z}_{\mathrm{p}}^{T}\mathbf{z}_{\mathrm{r}}}{|\mathbf{z}_{\mathrm{p}}||\mathbf{z}_{\mathrm{r}}|},$
(2)

where $|\cdot|$ denotes the Euclidean norm. The probe face is considered to
match the reference face if their similarity score exceeds a predefined
threshold $\tau$:

$\displaystyle\text{match}(\mathbf{z}_{\mathrm{p}},\mathbf{z}_{\mathrm{r}})=\begin{cases}1,&\text{if
}s(\mathbf{z}_{\mathrm{p}},\mathbf{z}_{\mathrm{r}})\geq\tau,\\\
0,&\text{otherwise}.\end{cases}$ (3)

The learned face recognition models are commonly assessed through their
verification (1:1 comparison) performance. For verification, metrics such as
true acceptance rate (TAR) at a fixed false acceptance rate (FAR) are
reported. In this paper, we report the TAR at $\mathrm{FAR}=0.01\%$.

Given the above system and protocol, the next section will empirically
demonstrate the impact of watermarking on the performance of face recognition
models trained using watermarked face images.

Figure 5:  Visualization of original images, watermarked images, and their
10$\times$ pixel differences for both real and synthetic face image datasets,
CASIA-WebFace [21] and DCFace [20], respectively. The pixel-wise difference is
color-inverted for enhanced visualization, where white areas denote zero
values.

##  4 Experiments

In this section, we demonstrate the quality of watermarked face images and
their robustness to diverse image transformations. Next, we assess the impact
of watermarking face images on the performance of given, pre-trained face
recognition models. Finally, we investigate the effect of using watermarked
face images to train face recognition models on downstream face recognition
tasks.

Table 1: Summary of face recognition datasets used in this study.

Dataset | # of IDs | # of Images |  | Avg. #
---
(Images/ID)
MS1Mv2 [57] | 85,742 | 5.8M | 68
CASIA-WebFace [21] | 10,572 | 0.5M | 46
DCFace [20] | 60,000 | 1.3M | 22

###  4.1 Experimental Setup

First, in Table 1, we provide a description of the datasets utilized in this
study, including the number of unique face identities, the total number of
images, and the average number of images per identity. For evaluation
purposes, we select a subset of 2 images per identity from all 10,547
identities in CASIA-WebFace and 2 images per identity from a randomly sampled
set of 10,000 identities for DCFace to compute genuine and imposter scores.
During training experiments, we utilize the full MS1Mv2 dataset.

Second, we implement the watermarking system following Sec. 3.1 and the
architecture setup of HiDDeN [17]. The watermark encoder consists of 4 Conv-
BN-ReLU blocks, each with 64 conv filters of 3$\times$3 kernels, 1 stride, and
1 padding. The decoder has 7 such blocks, followed by another block with $L$
filters ($L$ is the watermarking bit string length) and a linear layer of
$L$$\times$$L$. The watermark encoder and decoder are pre-trained together on
the whole MS-COCO dataset [62], cropped to $256\times 256$. The watermarking
performance will be evaluated on both CASIA-WebFace [21] and DCFace [20]
datasets.

Furthermore, to assess the impact of watermarking on face recognition model
performance, we employ three state-of-the-art face recognition models
pretrained on large, non-watermarked face recognition datasets. Specifically,
we consider two variations (ResNet50 and ResNet101) of AdaFace [63] pretrained
on MS1Mv2, along with one commercial-off-the-shelf (COTS) face recognition
SDK111This COTS SDK is among the top performers in the NIST FRVT Ongoing
competition. https://pages.nist.gov/frvt/html/frvt11.html. We evaluate the
performance on both the original CASIA-WebFace and DCFace datasets, as well as
their watermarked versions. For evaluation, we denote the face recognition
performance on the original dataset as ‘original-original’, indicating that
both the probe and reference face images being compared are original images.
‘Watermarked-original’ denotes that the probe image is watermarked while the
reference image is unmodified, and ‘watermarked-watermarked’ indicates that
both the probe and reference images are watermarked.

Finally, to explore the impact of watermarking on face recognition model
training, we train our own ResNet50 AdaFace models on both the original MS1Mv2
dataset and its watermarked version. We then compare the performance of these
trained models on the CASIA-WebFace dataset. Similarly, we conduct the same
experiment by training on both the original DCFace synthetic face image
dataset and its watermarked version to investigate the effect of watermarking
on synthetic face images.

###  4.2 Experimental Results

#### Assessing watermarking quality in face images.

In Fig. 5, we present visualizations of original images, watermarked images,
and pixel-wise differences between the original and watermarked images for
both real and synthetic face image datasets (CASIA-WebFace and DCFace,
respectively). Notably, the watermarked images exhibit no visually perceptible
differences compared to the originals, with pixel-wise discrepancies primarily
located along image edges, which are challenging for humans to discern.

To further evaluate this, we randomly selected 5 pairs of original/watermarked
images and solicited feedback from 10 student volunteers to differentiate
between them. Among the total 50 responses received, 8 were correct
selections, 6 were incorrect selections, and 36 were labeled as ‘Not sure’
(See Fig. 6). This underscores the invisibility of the watermark (the ‘$S$’
letter used in Fig. 3) to human eyes, indicating the stealthiness of
watermarking and its high quality.

Figure 6:  Survey for differentiating original/watermarked images.

Table 2 extends the analysis from Fig. 4, evaluating the watermark’s
resilience against different post-watermarking transformations (cropping,
resizing, brightness, contrast, and JPEG compression) at varying scaling
factors. For each transformation, scaling factors range from 1 to 0.75 for
cropping and resizing, 1 to 3.5 for brightness and contrast adjustments, and
100 to 75 for JPEG compression quality. As we can see, reconstructing the
watermark maintains high bit accuracy across diverse transformations and
scaling factors for both real (CASIA-WebFace) and synthetic (DCFace) datasets.
Notably, even when images are scaled down to 75% of their original size, bit
accuracy remains above 84% for cropping and resizing. Likewise, for brightness
and contrast adjustments, bit accuracy remains above 88% up to a scaling
factor of 3.5. However, JPEG compression has a more pronounced impact on
watermark extraction, with bit accuracy dropping to approximately 73% and 68%
for CASIA-WebFace and DCFace, respectively, at a quality factor of 75.

Table 2:  The robustness of watermarking evaluated using the reconstructed
watermark bit accuracy (%) against various data transformations at different
scaling strengths. Each value is averaged over 2000 images from each dataset,
with an image size of 112 $\times$ 112 and a watermark string bit length of
48.

Transformation | Dataset | Scaling ratio
---|---|---
1 | 0.95 | 0.9 | 0.85 | 0.8 | 0.75
Crop | CASIA-WebFace | 98.39 | 97.22 | 93.7 | 95.12 | 94.77 | 94.3
DCFace | 97.31 | 96.27 | 92.37 | 93.96 | 93.48 | 93.25
Resize | CASIA-WebFace | 98.39 | 92.47 | 92.0 | 91.58 | 89.62 | 85.93
DCFace | 97.31 | 90.24 | 89.99 | 89.64 | 87.09 | 84.32
Transformation | Dataset | Scaling factor
1 | 1.5 | 2 | 2.5 | 3 | 3.5
Brightness | CASIA-WebFace | 98.39 | 98.48 | 96.65 | 94.21 | 91.6 | 88.87
DCFace | 97.31 | 98.55 | 96.66 | 94.15 | 91.3 | 88.38
Contrast | CASIA-WebFace | 98.39 | 98.81 | 98.15 | 96.82 | 94.92 | 92.62
DCFace | 97.31 | 98.87 | 98.76 | 98.14 | 97.11 | 95.62
Transformation | Dataset | JPEG quality factor
100 | 95 | 90 | 85 | 80 | 75
JPEG compression | CASIA-WebFace | 98.39 | 90.36 | 85.0 | 80.8 | 76.65 | 73.06
DCFace | 97.31 | 87.65 | 80.02 | 75.34 | 71.75 | 68.53

Figure 7:  Watermarking robustness to different data transformations across
various transformation factors. The setup follows Table 2.  Table 3: Pre-
trained face recognition performance, TAR (%) at FAR=0.01%, on original images
vs. watermarked images. In the prob-preference pairs, ‘original-original’
indicates that both probe and reference face images to be compared are
original images. ‘Watermarked-original’ represents that the probe image is
watermarked while the reference image is unmodified, and ‘watermarked-
watermarked’ signifies that both the probe and reference images are
watermarked.

Evaluation Dataset | Probe-Reference | AdaFace ResNet501 | AdaFace ResNet1011 | COTS SDK
---|---|---|---|---
CASIA-WebFace (10,572 IDs, 2 images/ID) | Original-Original | 86.94 | 87.85 | 89.22
Watermarked-Watermarked | 86.36 | 87.62 | 88.68
Watermarked-Original | 86.61 | 86.61 | 88.66
DCFace (10,000 IDs, 2 images/ID) | Original-Original | 76.10 | 74.99 | 86.84
Watermarked-Watermarked | 73.65 | 71.83 | 84.56
Watermarked-Original | 74.83 | 74.83 | 85.47

  * •

1pre-trained on MS1MV2 dataset

Fig. 7 depicts the averaged bit accuracy of watermark reconstruction from
watermarked face images, based on the CASIA-WebFace and DCFace datasets
respectively, across different scaling factors for each data transformation
type. Overall, the bit accuracy on DCFace is slightly lower than that on
CASIA-WebFace, likely due to differences in image characteristics between real
and synthetic faces, given that the watermarking model is pre-trained on real
images. Nevertheless, the watermarking scheme demonstrates robust resilience
to various transformations on both datasets. The consistently high bit
accuracy across different scaling factors indicates that the watermark can be
reliably extracted even when watermarked images undergo common image
processing operations. This underscores the robustness and practicality of the
watermarking approach for protecting and authenticating face images.

#### Watermarking effect on pre-trained face recognition models.

Utilizing three SOTA face recognition systems, including two open-source
variants (ResNet50 and ResNet101 variants of AdaFace [63]), and a third from a
COTS face recognition SDK, we assess the impact of watermarking on face
recognition performance. Table 3 illustrates the recognition performance on
two face image datasets: CASIA-WebFace real face images and DCFace synthetic
face images. As we can see, watermarking face images results in a marginal
decrease in face recognition performance, as measured by True Accept Rate
(TAR) at a False Accept Rate (FAR) of 0.01%. For instance, the average
performance drop from ‘original-original’ to ‘watermarked-original’ across the
three matchers on the CASIA-WebFace dataset was 0.51%. This decline in
performance was more pronounced when both the probe and reference face images
were watermarked, resulting in a TAR drop of 0.81%. Furthermore, the impact of
watermarking was more substantial for the synthetic DCFace dataset, leading to
an average decrease of 1.15% when the probe image was watermarked and 3.35%
when both the probe and reference images were watermarked. The heightened drop
in performance on synthetic images due to the watermark raises concerns
regarding the use case of applying a watermark to generated face images to
maintain provenance, highlighting the need for further methods to embed the
watermark without impeding face recognition matching performance.

Figure 8: ResNet101 AdaFace model pre-trained on MS1Mv2 evaluated on (a)
CASIA-WebFace and (b) DCFace.  Figure 9: Examples of (a) success and (b)
failure cases using pre-trained ResNet101 AdaFace on CASIA-WebFace (similarity
score is shown for each pair).

To better understand the effect of watermarking on face recognition
performance, Fig. 8 presents the genuine and imposter similarity score
distributions for both CASIA-WebFace and DCFace datasets using the pre-trained
ResNet101 AdaFace model before and after applying the watermark to each probe
image. As we can see, the imposter score distribution is statistically
unchanged whereas the mean of the genuine score distribution is slightly
shifted left (from 0.502 to 0.491 for CASIA-WebFace and from 0.313 to 0.303
for DCFace) with the inclusion of the watermarked images. As a result, some
challenging comparisons near the match threshold for the original images may
be pushed to below the genuine match threshold when watermarked, slightly
degrading the performance of the face recognition model. This observation is
further supported by examples of the failure cases shown in Fig. 9. These
examples showcase challenging genuine comparisons that were matched slightly
above the threshold for the original images but resulted in false rejections
when the probe image was watermarked. And these rejections appear to align
with matching against challenging poses and expressions. To validate the
statistical significance of the effect of watermarking on the genuine
distributions, we conducted a two-sided T-test between the genuine
distributions for both datasets before and after applying the watermark. Both
results were statistically significant, yielding a p-value of $1.45\times
10^{-4}$ and $7.15\times 10^{-8}$ for CASIA-WebFace and DCFace, respectively.

Table 4: Effect of watermarking on face recognition training. Trained model
performance, in terms of TAR (%) at FAR = 0.01%, evaluated on CASIA-WebFace
and CASIA-WebFace of watermarked images.

Train Evaluation |  | CASIA-WebFace
---
(original)
| CASIA-WebFace
---
(watermarked)
| DCFace
---
54.26 | 49.30
| DCFace (watermarked)
---
53.64 | 51.67
| MS1Mv2
---
84.34 | 83.35
| MS1Mv2 (watermarked)
---
83.69 | 83.12

#### Effect of watermarking on the utility of face images for training face
recognition models.

In an ideal scenario, watermarking face images, whether real or synthetic,
should not impact downstream face recognition tasks. To evaluate the utility
of watermarked face images, we trained multiple face recognition models on
both non-watermarked and watermarked versions of the MS1Mv2 and DCFace
datasets. We then compared the accuracy of the trained models on the CASIA-
WebFace dataset. The resulting face recognition performance is presented in
Table 4. As we can see, there exists a slight decrease in recognition
performance when training on watermarked face images compared to training on
the corresponding original face images. For instance, training on MS1Mv2 with
and without the watermark led to a drop in recognition accuracy on CASIA-
WebFace from a TAR of 84.34% to 83.69%. Similarly, a similar trend was
observed when training on the DCFace image dataset and its watermarked
version, reducing the TAR on CASIA-WebFace from 54.26% to 53.64%.

In the previous experiment (Table 3), we noted a decrease in the performance
of pre-trained recognition models evaluated on the watermarked version of
CASIA-Webface vs. non-watermarked CASIA-Webface. Interestingly, from Table 4,
we can see that besides the drop in absolute performance compared to training
on non-watermarked images (row 1 vs. 2 and row 3 vs. 4), even if the
recognition model is trained on watermarked images (as is the case for rows 2
and 4), there is still a gap at test time evaluating on the watermarked
version of CASIA-WebFace compared to the non-watermarked CASIA-WebFace
(comparing column 2 to column 3); Albeit, this gap is smaller compared to the
models only trained on non-watermarked images (rows 1 and 3). Therefore, not
only does training on watermarked images decrease the absolute performance of
the face recognition model, it also does not provide total robustness to
watermarks during test time.

##  5 Conclusion

In this study, we conducted a thorough examination of the effects of digital
watermarking on face recognition, emphasizing the complexities and potential
benefits associated with the ethical utilization of generated face images. Our
devised framework, amalgamating face generation, watermarking, and
recognition, facilitated a methodical exploration of their interconnected
dynamics. Through rigorous experimentation, we showcased the efficacy of the
watermarking scheme in ensuring robust face image attribution, while also
shedding light on the inherent trade-offs between watermarking and recognition
accuracy. Our results highlight the imperative for future research endeavors
aimed at devising techniques capable of optimizing both watermarking and face
recognition performance concurrently. Furthermore, the adversarial robustness
of watermarking remains inadequately explored and warrants increased research
attention in the future. While our study concentrated on assessing the
robustness of watermarking against image transformations, its susceptibility
to adversarial attacks remains an open concern. Adversarial attacks could
potentially render the watermark removable, underscoring the necessity for
further investigation in this area.

##  6 Broader Impacts

Our investigation into the impact of digital watermarking on face recognition
holds broader implications for the responsible advancement of AI systems,
particularly in the domain of face recognition and biometrics, as well as in
the broader realm of generated AI. With the escalating sophistication of
generative models in producing lifelike face images, there arises a pressing
need to establish measures ensuring the integrity, authenticity, and
accountability of these generated images. Exploration in this work could shed
light on the potential of digital watermarking as a pivotal tool for image
attribution and ownership verification, pivotal in combatting the misuse of
generated images for nefarious purposes like deepfake creation and identity
fraud. Nevertheless, our findings also underline the imperative of carefully
balancing the trade-offs between watermarking and recognition performance.
While watermarking offers advantages in terms of image accountability, its
presence may impede the accuracy of face recognition systems under certain
circumstances. This highlights the necessity of developing techniques that
effectively reconcile these competing objectives, ensuring that the benefits
of watermarking can be realized without compromising the performance of
downstream applications. We aspire that our findings will catalyze further
research and stimulate discussions on how the responsible and ethical design
of generative AI contribute to broad AI applications.

## References

  * [1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [4] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522–22531, 2023.
  * [5] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu.  To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now.  arXiv preprint arXiv:2310.11868, 2023.
  * [6] Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li.  Protecting world leaders against deep fakes.  In CVPR workshops, volume 1, page 38, 2019.
  * [7] Bobby Chesney and Danielle Citron.  Deep fakes: A looming challenge for privacy, democracy, and national security.  Calif. L. Rev., 107:1753, 2019.
  * [8] Zhangheng Li, Junyuan Hong, Bo Li, and Zhangyang Wang.  Shake to leak: Fine-tuning diffusion models can amplify the generative privacy risk.  arXiv preprint arXiv:2403.09450, 2024.
  * [9] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253–5270, 2023.
  * [10] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Understanding and mitigating copying in diffusion models.  Advances in Neural Information Processing Systems, 36:47783–47803, 2023.
  * [11] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Diffusion art or digital forgery? investigating data replication in diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048–6058, 2023.
  * [12] Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu.  Unlearncanvas: A stylized image dataset to benchmark machine unlearning for diffusion models.  arXiv preprint arXiv:2402.11846, 2024.
  * [13] Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite.  Stable bias: Evaluating societal representations in diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [14] Yue Jiang, Yueming Lyu, Tianxiang Ma, Bo Peng, and Jing Dong.  Rs-corrector: Correcting the racial stereotypes in latent diffusion models.  arXiv preprint arXiv:2312.04810, 2023.
  * [15] Mingyuan Fan, Cen Chen, Chengyu Wang, and Jun Huang.  On the trustworthiness landscape of state-of-the-art generative models: A comprehensive survey.  arXiv preprint arXiv:2307.16680, 2023.
  * [16] Anil K Jain and Umut Uludag.  Hiding biometric data.  IEEE transactions on pattern analysis and machine intelligence, 25(11):1494–1498, 2003.
  * [17] Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei.  Hidden: Hiding data with deep networks.  In Proceedings of the European conference on computer vision (ECCV), pages 657–672, 2018.
  * [18] Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon.  The stable signature: Rooting watermarks in latent diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22466–22477, 2023.
  * [19] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein.  Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust.  arXiv preprint arXiv:2305.20030, 2023.
  * [20] Minchul Kim, Feng Liu, Anil Jain, and Xiaoming Liu.  Dcface: Synthetic face generation with dual condition diffusion model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12715–12725, 2023.
  * [21] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li.  Learning face representation from scratch.  arXiv preprint arXiv:1411.7923, 2014.
  * [22] Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, et al.  Webface260m: A benchmark unveiling the power of million-scale deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10492–10502, 2021.
  * [23] Yichun Shi, Debayan Deb, and Anil K Jain.  Warpgan: Automatic caricature generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10762–10771, 2019.
  * [24] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  arXiv preprint arXiv:2308.06721, 2023.
  * [25] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan.  Photomaker: Customizing realistic human photos via stacked id embedding.  arXiv preprint arXiv:2312.04461, 2023.
  * [26] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen.  Instantid: Zero-shot identity-preserving generation in seconds.  arXiv preprint arXiv:2401.07519, 2024.
  * [27] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [28] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [29] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789–8797, 2018.
  * [30] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in neural information processing systems, 32, 2019.
  * [31] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [32] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018.
  * [33] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  Advances in neural information processing systems, 33:12104–12114, 2020.
  * [34] Vahid Mirjalili, Sebastian Raschka, Anoop Namboodiri, and Arun Ross.  Semi-adversarial networks: Convolutional autoencoders for imparting privacy to face images.  In 2018 International Conference on Biometrics (ICB), pages 82–89. IEEE, 2018.
  * [35] Mika Westerlund.  The emergence of deepfake technology: A review.  Technology innovation management review, 9(11), 2019.
  * [36] Ingemar Cox, Matthew Miller, Jeffrey Bloom, Jessica Fridrich, and Ton Kalker.  Digital watermarking and steganography.  Morgan kaufmann, 2007.
  * [37] Matthew L Miller, Ingemar J Cox, and Jeffrey A Bloom.  Informed embedding: exploiting image and detector information during watermark insertion.  In Proceedings 2000 International Conference on Image Processing (Cat. No. 00CH37101), volume 3, pages 1–4. IEEE, 2000.
  * [38] Sviatoslav Voloshynovskiy, Shelby Pereira, Thierry Pun, Joachim J Eggers, and Jonathan K Su.  Attacks on digital watermarks: classification, estimation based attacks, and benchmarks.  IEEE communications Magazine, 39(8):118–126, 2001.
  * [39] Jessica Fridrich.  Visual hash for oblivious watermarking.  In Security and Watermarking of Multimedia Contents II, volume 3971, pages 286–294. SPIE, 2000.
  * [40] Alessandro Piva, Mauro Barni, Franco Bartolini, and Vito Cappellini.  Dct-based watermark recovering without resorting to the uncorrupted original image.  In Proceedings of international conference on image processing, volume 1, pages 520–523. IEEE, 1997.
  * [41] Mauro Barni, Franco Bartolini, Vito Cappellini, and Alessandro Piva.  A dct-domain system for robust image watermarking.  Signal processing, 66(3):357–372, 1998.
  * [42] Xiang-Gen Xia, Charles G Boncelet, and Gonzalo R Arce.  A multiresolution watermark for digital images.  In Proceedings of international conference on image processing, volume 1, pages 548–551. IEEE, 1997.
  * [43] Ruizhen Liu and Tieniu Tan.  An svd-based watermarking scheme for protecting rightful ownership.  IEEE transactions on multimedia, 4(1):121–128, 2002.
  * [44] Kevin Alex Zhang, Alfredo Cuesta-Infante, Lei Xu, and Kalyan Veeramachaneni.  Steganogan: High capacity image steganography with gans.  arXiv preprint arXiv:1901.03892, 2019.
  * [45] Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz.  Artificial fingerprinting for generative models: Rooting deepfake attribution in training data.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 14448–14457, 2021.
  * [46] Jie Zhang, Dongdong Chen, Jing Liao, Weiming Zhang, Huamin Feng, Gang Hua, and Nenghai Yu.  Deep model intellectual property protection via deep watermarking.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4005–4020, 2021.
  * [47] Vlasislav Skripniuk.  Watermarking for generative adversarial networks.  PhD thesis, Universität des Saarlandes Saarbrücken, 2020.
  * [48] Hanzhou Wu, Gen Liu, Yuwei Yao, and Xinpeng Zhang.  Watermarking neural networks with watermarked images.  IEEE Transactions on Circuits and Systems for Video Technology, 31(7):2591–2601, 2020.
  * [49] Jianwei Fei, Zhihua Xia, Benedetta Tondi, and Mauro Barni.  Supervised gan watermarking for intellectual property protection.  In 2022 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6. IEEE, 2022.
  * [50] Matthew Turk and Alex Pentland.  Eigenfaces for recognition.  Journal of cognitive neuroscience, 3(1):71–86, 1991.
  * [51] Md Abdur Rahim, Md Najmul Hossain, Tanzillah Wahid, and Md Shafiul Azam.  Face recognition using local binary patterns (lbp).  Global Journal of Computer Science and Technology, 13(4):1–8, 2013.
  * [52] Alberto Albiol, David Monzo, Antoine Martin, Jorge Sastre, and Antonio Albiol.  Face recognition using hog–ebgm.  Pattern Recognition Letters, 29(10):1537–1543, 2008.
  * [53] Harihara Santosh Dadi and GK Mohan Pillutla.  Improved face recognition rate using hog features and svm classifier.  IOSR Journal of Electronics and Communication Engineering, 11(04):34–44, 2016.
  * [54] Pallabi Parveen and Bhavani Thuraisingham.  Face recognition using multiple classifiers.  In 2006 18th IEEE International Conference on Tools with Artificial Intelligence (ICTAI’06), pages 179–186. IEEE, 2006.
  * [55] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701–1708, 2014.
  * [56] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [57] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [58] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [59] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [61] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 87–102. Springer, 2016.
  * [62] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [63] Minchul Kim, Anil K Jain, and Xiaoming Liu.  Adaface: Quality adaptive margin for face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18750–18759, 2022.
  * [64] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han.  Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices.  In Biometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings 13, pages 428–438. Springer, 2018.
  * [65] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265–5274, 2018.