# Data-Free Knowledge Distillation Using Adversarially
Perturbed OpenGL Shader Images

Logan Frank Jim Davis
Department of Computer Science and Engineering
Ohio State University
{frank.580<EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) has been a popular and effective method for model
compression. One important assumption of KD is that the original training
dataset is always available. However, this is not always the case due to
privacy concerns and more. In recent years, “data-free” KD has emerged as a
growing research topic which focuses on the scenario of performing KD when no
data is provided. Many methods rely on a generator network to synthesize
examples for distillation (which can be difficult to train) and can frequently
produce images that are visually similar to the original dataset, which raises
questions surrounding whether privacy is completely preserved. In this work,
we propose a new approach to data-free KD that utilizes unnatural OpenGL
images, combined with large amounts of data augmentation and adversarial
attacks, to train a student network. We demonstrate that our approach achieves
state-of-the-art results for a variety of datasets/networks and is more stable
than existing generator-based data-free KD methods. Source code will be
available in the future.

##  1 Introduction

Neural networks have become a dominant force in machine learning (ML) [25, 58,
16]. Enabled by modern compute hardware, the size of the largest available
networks has continued to increase with time [33, 14, 32]. Coinciding with the
desire to increase model capacity, there has been a growing interest in
deploying small, but well-performing, networks on edge devices [59, 51].
Utilizing neural networks on such devices is often difficult as there are
typically strict resource constraints such as memory or power consumption.
Unfortunately, models frequently have to sacrifice performance in order to
satisfy these imposed limitations. This often points practitioners towards
considering existing methods that enable a smaller network to perform as well
as a larger network all while retaining its lean characteristic.

Paradigms that address the aforementioned goal are called model compression
techniques, which as the name implies, aim to compress a large and complex
neural network that performs well on some downstream task (classification,
object detection, etc.) into one with a more compact and efficient form. One
such method is knowledge distillation (KD) [27], where the goal is to transfer
the information stored inside a cumbersome “teacher” network to a more compact
“student” network (which can be a completely different architecture). This is
accomplished by utilizing the soft-target outputs (or internal features) of
the teacher network to guide the student towards mimicking the output
responses of the teacher when presented with similar inputs. It has even been
shown that performance gains can be achieved by distilling between a teacher
and student that are the exact same network architecture (self-distillation)
[19]. Other works have built upon the standard KD approach by applying
adversarial attacks to the original data for exploiting decision boundary
information in the teacher model, which resulted in improved student accuracy
[26, 55].

Figure 1: Example TwiGL OpenGL shader images.

While KD has been a remarkably popular mechanism for creating lightweight
models, one particular drawback that it (along with other model compression
techniques [29, 21] and KD derivatives [26, 55]) suffers from is that standard
KD relies on the strong assumption that the original training data (or similar
distribution data) will be available at distillation. However, this is not
often the case as privacy concerns continue to grow regarding the use and
transferring of data. Moreover, the data could be kept as proprietary in-house
information by a company or simply put, the dataset is too large to transfer
or store in a reasonable manner. To circumvent this drawback, many approaches
have been proposed in the realm of data-free knowledge distillation (DFKD),
which utilizes the teacher as a means to generate synthetic examples to
orchestrate the distillation process. These methods typically involve
generating examples by either sampling from distributions modeled off prior
information contained inherently in the teacher (_e.g_., batch norm [28]
statistics) [40, 60] or an additional generator network [22] trained
concurrently with distillation [9, 11, 4]. Incidentally, the synthetic
examples generated by these approaches often closely resemble or give insights
to the original data [64, 65]. This begs the question as to whether or not
these DFKD approaches are truly privacy-preserving. Furthermore, the potential
inclusion of a generator network can complicate the KD process as there is a
completely different set of hyperparameters to be managed as well as the
potential issues of mode collapse or non-convergence [48].

In this work, we present a novel approach to DFKD that differs from existing
methods in that it does not utilize an additional generator network or attempt
to extract naturalistic synthetic examples from the teacher. Instead, we
propose to 1) construct a dataset of unnatural synthetic images using OpenGL
shaders, 2) adversarially perturb them to identify the decision boundaries in
the teacher model, and 3) transfer this knowledge to the student using
standard KD [27]. Experiments show significant improvements over existing DFKD
approaches for multiple datasets and network architectures, all while being
completely privacy-preserving with respect to the original dataset. We
additionally show qualitatively that our distilled models can better anonymize
the original data embedded in the teacher. Our contributions are summarized as
follows:

  1. 1.

A new framework for data-free knowledge distillation that exploits synthetic
imagery and adversarial perturbations rather than utilizing a generator
network.

  2. 2.

The proposed approach utilizes the standard knowledge distillation regime and
can easily be incorporated into more advanced distillation techniques.

  3. 3.

Our method is completely privacy-preserving to the original dataset and
furthermore enables a path for anonymizing the data embedded in the teacher
model.

We begin with a review of related work in Sect. 2. The components of our
proposed DFKD approach are described in Sect. 3. Lastly, extensive experiments
demonstrating our method are presented in Sect. 4.

##  2 Related Work

In recent years, many works have been proposed in the areas of KD, DFKD, ML
privacy, and adversarial examples.

Knowledge Distillation. The transferring of knowledge from a large network to
a smaller network was introduced in [5] and was further refined and coined as
“knowledge distillation” in [27]. The general KD framework outlined in [27]
trains a student network to match the temperature-scaled [23] soft outputs
from a larger teacher network using entropy-based loss functions. Since then,
several works have investigated what properties influence the success of KD
[57, 10, 3] and others have proposed structural improvements to the seminal
approach [45, 49, 54, 53]. Notably, [3] argued that KD can be viewed as
“function matching” and showed that applying mixup [66] to the input
distillation images results in improved student performance. They also
observed that some knowledge can be transferred to a student network using
out-of-domain data, albeit at a significant performance loss compared to
employing in-domain data. Furthermore, KD has been coupled with other model
compression techniques [39] and applied to other areas such as semi-supervised
learning [62], multi-exit architectures [42], and more. Rather than distilling
from a large cumbersome network to a smaller network, self-distillation
repeatedly transfers knowledge to a student that is the same architecture as
the teacher [19].

Data-Free Knowledge Distillation. In standard KD, it is assumed that the
original training dataset is available. However this is not always the case,
which has motivated a series of data-free KD approaches that attempt to
transfer knowledge when no data is available. This line of work can be
separated into two categories based on whether they utilize a generator
network [9, 65, 11, 18, 4] or inherent teacher network statistics [40, 60, 64]
to synthesize examples that may be beneficial for KD. The first truly data-
free approach was proposed in [40], where they modelled Dirichlet
distributions at the output of the teacher and synthesized examples to match
these distributions using backpropagation. In [9], the teacher is treated as a
fixed discriminator while a generator is trained to output images that
maximize the one-hot cross entropy loss for the teacher. A similar concept is
proposed in [11] with an added constraint that minimizes the divergence
between the generated example’s features and the teacher’s learned batch norm
layer [28] statistics.

Later, we will compare to the relevant approaches of Contrastive Model
Inversion (CMI) [18] and Pseudo Replay Enhanced DFKD (PRE) [4]. Both CMI and
PRE rely on a generator network to synthesize examples for transferring
knowledge from teacher to student. More specifically, CMI utilizes the
framework presented in [11], which consists of three different loss
components, with accompanied hyperparameter weight coefficients, that guide
the generator’s outputs. They introduced a novel contrastive loss component in
addition to the aforementioned losses that forces the generator to output new
examples that differ from previously synthesized examples that are stored in a
memory bank. As for PRE, they utilize the approach outlined in [9], consisting
of four loss terms (two of which are similar to CMI), also associated with
multiple hyperparameter weight coefficients, used to train the generator
network. Rather than using a memory bank, PRE utilized an additional
variational autoencoder [31] to remember previously synthesized examples
created by the novel-view generator network.

Our work differs significantly from previous methods in that we do not utilize
an additional generator network nor do we attempt to synthesize naturalistic
examples using the teacher. We do leverage the teacher to obtain information
about its decision boundaries, however the samples we obtain remain unnatural
in appearance.

ML Privacy. In recent years, data privacy and the inherent privacy of ML
models have become major concerns. It has been shown that private training
examples can be extracted from MLP networks [24], large language models [7],
and diffusion models [6]. Although DFKD is motivated by privacy issues
surrounding the original dataset, many approaches can generate examples that
are visually similar or have artifacts similar to examples from the original
data [64, 9, 65, 11, 18]. This raises an argument as to whether privacy is
completely preserved in these methods.

Adversarial Examples. Another major concern for neural networks is adversarial
examples [52]. These are examples that have been intentionally perturbed to
appear unchanged, but have been manipulated by an adversarial attack to be
classified by a model as something different (_e.g_., an image of a “dog” that
was perturbed and classified as “guacamole”). Numerous adversarial attacks
have been proposed with varying benefits and capabilities [52, 35, 38, 8]. To
combat these attacks, several defenses [41, 61, 44] and detection [36, 1]
methods have been introduced over the years. Rather than attempting to be
robust against or detect such inputs, adversarial examples were shown to
provide accuracy improvements for standard KD in [26, 55]. These approaches
used adversarial attacks on real data to create slightly modified examples
that helped identify the decision boundaries in the teacher network. We employ
adversarial attacks for a similar goal, however we utilize pairs of
adversarial examples to better outline the decision boundary and also include
an additional stronger/deeper attack. Furthermore, only standard KD with the
original in-domain training dataset is considered in [26, 55], whereas we
utilize synthetic out-of-domain examples in a different KD regime where access
to the original data is prohibited.

##  3 Method

In this section, we describe the main components of our approach: 1) the
creation of a synthetic dataset, 2) the role of data augmentation on that
dataset, 3) how to utilize adversarial perturbations on the dataset to
identify the decision boundaries of the teacher, and lastly, 4) the method for
transferring knowledge from teacher to student.

###  3.1 OpenGL Shader Image Dataset

Our focus for this work is to transfer knowledge from a pretrained network to
a freshly initialized network without accessing the original dataset. We
instead want to create some new dataset with the added constraints that there
is no additional generator network used for synthesizing these images nor any
other mechanism for attempting to recover “desirable” images from the teacher.
Therefore, we leverage procedural image programs (using OpenGL [50]) for
rendering synthetic images to construct the dataset we will use for KD.
Utilizing the approach of [2], we first synthesize several images for each of
the available 1089 TwiGL shaders [56]. Examples of these shader images are
shown in Fig. 1. As some of these shaders produced images that were either
constant (_i.e_., containing all one color such as all black or all white) or
simple (_i.e_., containing only two colors or containing few colored pixels),
we filter the rendered shader images in order create an initial set of
synthetic images with the most diversity possible by removing the
aforementioned images. With this scheme, we can synthesize near-infinite
amounts of unnatural, out-of-domain images to construct a dataset that can be
leveraged for KD as will be shown.

Given a teacher network $\mathcal{F}_{t}$ pretrained on some dataset with
$\mathcal{C}=\\{c_{1},...,c_{R}\\}$ classes and a filtered synthetic dataset
$\mathcal{D}_{S}$, we pass every example in $\mathcal{D}_{S}$ through
$\mathcal{F}_{t}$ to obtain an initial teacher prediction and aggregate all
predictions. To form the final synthetic dataset $\mathcal{D}_{K}$ that will
be used for KD, we randomly select examples from $\mathcal{D}_{S}$ based on
their teacher predictions. For each class $c_{i}$ in the teacher’s dataset, we
uniformly sample examples from $\mathcal{C}_{-i}$ (the set of classes
$\mathcal{C}$ without $c_{i}$) to meet a desired number of synthetic images
$N_{i}$ per $c_{i}$ where

$\displaystyle\mathcal{C}_{-i}$ $\displaystyle=\\{\ c_{j}\ \ \forall\ \
j\in\mathcal{C}\ \ \ \ s.t.\ \ \ \ j\neq i\ \\}$ (1)

and the set of examples $X_{j}$ assigned to each $c_{j}\in\mathcal{C}_{-i}$
are

$\displaystyle X_{j}$ $\displaystyle=\\{\ x_{k}\ \ \forall\ \
x_{k}\in\mathcal{D}_{S}\ \ \ \ s.t.\ \ \ \ \mathcal{F}_{t}(x_{k})=c_{j}\ \\}$
(2)

In other words, we are selecting examples for $c_{i}$ that are predicted as
any other label besides $c_{i}$. If uniformly sampling across
$\mathcal{C}_{-i}$ does not meet the specified number of examples per class
(_i.e_., $N_{i}$ is not evenly divisible by $R-1$), then the remaining
examples are randomly sampled without replacement until $N_{i}$ examples are
obtained. For all images collected for class $c_{i}$, we assign them an
associated target label $t=c_{i}$. Note that we do allow repeats between
classes (_i.e_., a specific image can be assigned to both $c_{1}$ and
$c_{2}$).

Collecting $N_{i}$ shader images per class for the specific pretrained teacher
network creates the base synthetic distillation dataset
$\mathcal{D}_{K}=\\{(x_{1},t_{1}),...,(x_{N},t_{N})\\}$ where

$\displaystyle\mathcal{F}_{t}(x_{j})\neq t_{j}\ \ \forall\ \
(x_{j},t_{j})\in\mathcal{D}_{K}$ (3)

Later, we will discuss why it is important that all examples not be classified
as their associated target label by default, and further describe how
$\mathcal{D}_{K}$ is used for distilling knowledge from the associated
pretrained teacher network to a new, freshly initialized student network.

###  3.2 Data Augmentation

As mentioned, one of the benefits to our method for creating a synthetic
dataset is the ability to render an unbounded number of images per shader
code, enabling near-infinite dataset sizes if desired. However, it becomes
obvious that obtaining hundreds of thousands to potentially millions of images
quickly poses a storage issue. A clear solution for increasing dataset
diversity without having an over-inflated dataset is data augmentation. This
allows us to artificially create more examples during distillation and
furthermore, augmented examples could enable us to explore regions of the
teacher’s feature space that the original data could not discover on its own.

In ordinary fully-supervised training, data augmentations should be “label-
preserving” [20]. However, we do not have such constraint as there is no
notion of a “label” (or any other semantic meaning) with our OpenGL shader
images. Thus, we can employ large amounts of data augmentation that would
normally not be considered. Later, we will describe our complete data
augmentation regime for KD training and additionally show how data
augmentation has provided a strong benefit to our approach that would likely
not be obtained through more physical examples.

###  3.3 Decision Boundary Exploitation

To transfer the most amount of information possible to the student, we utilize
targeted adversarial attacks on our synthetic examples to identify the
decision boundaries in the teacher network. As will be described below, our
attack is based on the Basic Iterative Method (BIM) [35].

Beginning with a synthetic example $x_{j}$ having target label $t_{j}$ where
$\mathcal{F}_{t}(x_{j})\neq t_{j}$, the goal of our adversarial attack is to
perturb the image “from the outside in”, starting at an arbitrary label and
ending at a desired label $t_{j}$. This allows us to identify the decision
boundary between $t_{j}$ and all other classes in the teacher
$\mathcal{F}_{t}$. Thus, it is important that the augmented synthetic example
be initially classified by the teacher as any other label besides $t_{j}$.
Generally speaking, our attack will perturb an example until it has crossed
into the decision space of $t_{j}$ and is within some specified argmax softmax
threshold (to ensure the example is somewhat close to the boundary). In
addition to the final adversarial image obtained from a successful attack, we
also take the example from the iteration just before the final image crossed
the boundary. This creates a pair of examples consisting of the “post”-success
example (the final image) and “pre”-success example (the second-to-last
image). This pre-success image is also required to be within the specified
argmax softmax threshold, but be classified as anything besides $t_{j}$.

After conducting our adversarial attack on a batch of synthetic images, we
filter (remove) examples that do not meet either of the following criteria: 1)
image not classified as its assigned target at the end of the adversarial
attack or 2) either the final (post-success) image or the pre-success image
were unable to meet the softmax threshold. This is to ensure that the examples
we are using to guide distillation are beneficial in identifying some
meaningful part of the feature space in the teacher, such as the decision
boundary.

To aid in the adversarial attack process, we propose adding a Bold Driver
heuristic to the attack step size to make it more adaptive and increase its
ability to create adversarial examples that meet our conditions. Thus, if
after a particular attack iteration the example has not reached the desired
target label, the step size will be increased. If the example has crossed the
decision boundary to the desired label but the argmax softmax value is above
the specific threshold, the example will be reverted back to its previous
iteration and the step size will be decreased. Finally, if an example has met
the desired final conditions (correct classification label and below argmax
softmax threshold), but the pre-success counterpart does not meet the softmax
threshold condition, the image will be reverted to the previous iteration and
the step size will be decreased (to move the pre-success example closer to the
decision boundary).

Beyond just identifying the decision boundary in the teacher model, we find
that adding an additional attack, which targets the pair of boundary examples
to be perturbed “deeper” into the decision space of their respective classes,
leads to additional performance gains. This attack simply clones the boundary
examples and performs a targeted BIM attack, where the target labels are the
examples’ teacher predictions. We find that a single attack step with as
little as $\epsilon=1$ is sufficient, with no noticeable benefit to employing
a larger $\epsilon$ or more iterations. The full adversarial perturbation
process is shown in Fig. 2 for an arbitrary class $c$.

(a)

(b)

(c)

Figure 2: Decision boundary exploitation adversarial attack in the teacher
feature space for an arbitrary class $c$. The symbols $\times$, $\times$,
$\times$, and $\times$ represent the original synthetic examples,
“post”-success examples, “pre”-success examples, and “deeper” examples,
respectively. Best viewed in color.

With a curated synthetic dataset ready to be used for knowledge distillation
on a specific teacher network, and a method for altering the data to identify
the decision boundaries in the teacher, we will next discuss our approach for
distilling knowledge from the teacher to the student.

###  3.4 Knowledge Distillation

In this section, we describe our method for transferring knowledge from
teacher to student, which is based upon the standard approach for KD [27].
Given a pretrained teacher network $\mathcal{F}_{t}$ and a randomly
initialized student network $\mathcal{F}_{s}$, an example is first augmented
then perturbed using $\mathcal{F}_{t}$. After completing the adversarial
attack process, the newly perturbed example is passed forward through both
networks to produce the teacher and student softmax distributions $p_{t}$ and
$p_{s}$, respectively. The knowledge distillation loss is computed on these
softmax scores as

$\displaystyle\mathcal{L}(p_{t}\ ||\ p_{s})$ $\displaystyle=\sum_{i\ \in\
\mathcal{C}}\left[\ p_{t(i)}log\ p_{t(i)}-p_{t(i)}log\ p_{s(i)}\ \right]$ (4)

which is simply the KL-divergence. Like [27], we also include a temperature
parameter $\tau$ to adjust the entropy of the output softmax distributions
from the teacher and student networks before they are used to compute the loss
(_i.e_., $p_{t}\propto exp(\frac{log\ p_{t}}{\tau})$ and $p_{s}\propto
exp(\frac{log\ p_{s}}{\tau})$).

We find that by combining our teacher-specific synthetic dataset with heavy
data augmentation and our proposed adversarial attacks, knowledge can be
sufficiently transferred from a teacher to a student using the standard KD
approach.

##  4 Experiments

We first conducted experiments in a self-distillation setting, followed by a
typical large teacher to small student KD scenario. Next, we compared to other
relevant DFKD approaches, and then investigated the various components of our
approach via ablation and sensitivity studies. Lastly, we conducted a
qualitative privacy study on various models.

Datasets and Networks. We employed three established image classification
datasets for our evaluation: CIFAR10 (C10) [34], CIFAR100 (C100) [34], and
Tiny ImageNet (Tiny ImgNet) [15], which contain 10, 100, and 200 classes,
respectively. For all datasets, we randomly sampled 10% of the training
examples class-wise for validation. The remaining training examples were used
to train the teacher networks, with the validation set used to select the best
model.

A variety of network architectures are considered for each of the datasets,
consisting of ResNets (RN) [25], Wide ResNets (WRN) [46], ResNeXts (RX) [63],
MobileNetV2 (MNv2) [47], and ShuffleNetV2 (SNv2) [37]. Specific variants for
each dataset will be discussed in the appropriate sections below. We trained
each selected network and dataset combination using cross entropy to establish
a baseline for each model and report those results in the tables.

Baseline Training Details. To train our teachers and reference baseline
models, we used SGD with momentum (0.9) and weight decay (1e-4) and a one-hot
cross-entropy loss. All networks were trained for 400 epochs with a batch size
of 256 and a half-period cosine learning rate scheduler, beginning with an
initial value of 0.1. Data augmentation consisted of RandAugment [12]
($n=2,m=14$), random horizontal flipping, and random cropping with padding.
The epoch yielding the best validation accuracy was used to select the final
model.

Distillation Training Details. For our DFKD experiments, we similarly employed
SGD with momentum (0.9) and weight decay (1e-4) for our student networks, and
utilized the standard KD loss ($\tau=20$) as defined previously [27].
Distillation was performed for 400 epochs for all datasets using a half-period
cosine learning rate scheduler with an initial value of 0.1, and batch sizes
of 128 for CIFAR10/100 and 64 for Tiny ImageNet. Our synthetic datasets
consisted of 25K OpenGL shader images for each of the datasets. We
additionally experimented with increasing the number of synthetic examples
used on CIFAR10 in our sensitivity studies. Our standard data augmentation
regime for KD consisted of RandAugment ($n=4,m=14$), random elastic transform,
random inversion, random horizontal flipping, and random cropping with
padding. Furthermore, we also included mixup [66] with
$\alpha\in\mathcal{U}(0,1)$, similar to [3]. As previously stated, we included
more augmentation for KD to add a large amount of data diversity into the
process.

The last major component of our approach is the adversarial attacks. The first
(“border”) attack for identifying decision boundaries in the teacher network
is a targeted $L_{\infty}$ BIM attack with $\epsilon=10$, an initial step size
of $\alpha=1$, and a maximum of 12 iterations. The softmax threshold for this
first attack is $0.95$, with other values considered later in our sensitivity
studies. After identifying the decision boundary and removing examples that
did not meet the filtering criteria (Sect. 3.3), the second (“deeper”)
adversarial attack is then applied, which is a targeted $L_{\infty}$ BIM
attack with $\epsilon=\alpha=1$ for a single step (no softmax threshold).

Evaluation. Three separate runs were performed for each distillation
experiment using sequential seed values of 1, 2, and 3, which were made more
complex using MD5 as suggested in [30, 43, 13]. The mean and standard
deviation of the three trials are reported. The student model from the last
epoch of distillation was selected for evaluation as validation checking
cannot be performed in this data-free context.

###  4.1 Results

The experimental results are shown in Table 1 for self-distillation, Table 2
for distilling from large teacher to smaller student, Table 3 for comparisons
to related approaches, and Table 4 for ablation and sensitivity studies. Our
qualitative privacy study is presented in Sect. 4.2.

Self-Distillation. To evaluate how well a network can transfer its knowledge
to another network that is the same architecture, we chose one smaller network
and one larger network for each dataset. This consisted of MNv2 and WRN22x8
for CIFAR10, RN18 and RX29-64x4 for CIFAR100, and RN18 and RX29-64x8 for Tiny
ImageNet. Results of this experiment are shown in Table 1. We see that for all
datasets and network architectures, knowledge can be transferred from a
pretrained network to a freshly initialized one using our approach, albeit at
a small performance loss (which is typical for DFKD). Our method still
achieves competitive performance levels, typically falling within 2% of the
network’s baseline accuracy. The accuracy degradation seen in our distilled
students can be lessened if we include more synthetic examples or train for
more epochs, as will be shown later. Furthermore, we will also revisit these
self-distillation models later in our privacy studies.

|  | Accuracy
---|---|---
Dataset | Model |  T / S |  T $\rightarrow$ S
C10 | MNv2 |  95.05 |  93.86$\pm$0.10
WRN22x8 |  96.21 |  95.18$\pm$0.09
C100 | RN18 |  74.30 |  72.56$\pm$0.23
RX29-64x4 |  77.82 |  76.67$\pm$0.04
Tiny ImgNet | RN18 |  61.81 |  60.59$\pm$0.08
RX29-64x8 |  64.16 |  62.09$\pm$0.24
Table 1: Accuracy on various datasets when self-distilling from two networks
of the same architecture using our approach.

Distilling Down. In most cases, KD is performed between a large, pretrained
teacher model and a much smaller, freshly initialized student. Here, we
employed teacher$\rightarrow$student pairs of RX29-64x4$\rightarrow$MNv2 and
WRN22x8$\rightarrow$RN18 for CIFAR10, RN50$\rightarrow$MNv2 and
RX29-64x4$\rightarrow$RN18 for CIFAR100, and WRN28x10$\rightarrow$SNv2 and
RX29-64x8$\rightarrow$RN18 for Tiny ImageNet. We additionally trained a
teacher RX50-32x8 model with mixup ($\alpha=0.2$) and distilled it to a RN18
for each of the datasets to investigate the behavior of our approach when an
even stronger teacher is employed. In the future, we will include one
teacher$\rightarrow$student pair for the standard ImageNet dataset. This
experiment will use BiT-M-ResNet 152x2 [33] as a teacher and RN50 as a
student, similar to [3], to demonstrate how our approach can scale up to
larger datasets. This network pair was trained for 600 epochs with 50K
synthetic OpenGL images. Pretrained weights for that teacher model were taken
from [3]. Results for all four datasets are reported in Table 2.

Knowledge is still able to be transferred from the large teacher to the small
student using our proposed method. However, we observe that there is a larger
gap between the baseline cross entropy student and the distilled student
compared to the self-distillation results. It is likely that, because of
either the change in network architecture or the complexity of the larger
teacher models, more synthetic examples, even stronger data augmentation, or
more distillation epochs could be needed to narrow the gap in performance.

|  |  | Accuracy
---|---|---|---
Dataset | Teacher | Student |  T |  S |  T $\rightarrow$ S
C10 | RX29-64x4 | MNv2 |  95.98 |  95.05 |  91.89$\pm$0.15
WRN22x8 | RN18 |  96.21 |  95.23 |  94.12$\pm$0.05
RX50-32x8 | RN18 |  96.90 |  95.23 |  93.93$\pm$0.14
C100 | RN50 | MNv2 |  77.44 |  76.15 |  71.99$\pm$0.13
RX29-64x4 | RN18 |  77.82 |  74.30 |  71.73$\pm$0.13
RX50-32x8 | RN18 |  81.54 |  74.30 |  72.92$\pm$0.18
Tiny ImgNet | WRN28x10 | SNv2 |  65.13 |  63.33 |  59.10$\pm$0.20
RX29-64x8 | RN18 |  64.16 |  61.81 |  56.05$\pm$0.22
RX50-32x8 | RN18 |  71.02 |  61.81 |  54.37$\pm$0.33
ImgNet | B-RN152x2 | RN50 |  85.15 |  80.86 |  TBD
Table 2: Accuracy on various datasets when distilling from a large teacher to
a smaller student using our approach.

Comparisons. Next, we compared to the existing DFKD methods of Contrastive
Model Inversion (CMI) [18] and Pseudo Replay Enhanced DFKD (PRE) [4]. We
include the reported RN34$\rightarrow$RN18 results of CMI and PRE from [4],
which used their own teacher models and optimal hyperparameter settings
(denoted as CMI* and PRE* in the table). However, for a more direct
comparison, we also ran the code for CMI/PRE with the same hyperparameter
settings and report those scores for our own teacher$\rightarrow$student pairs
as a dual-comparison (denoted as CMI and PRE in the table). When implementing
these approaches, we used their code available on GitHub and the
hyperparameters they specify for these datasets. Moreover, the authors only
provide one set of hyperparameter values which we used for all experiments. We
emphasize that we do not alter the code in any form besides simply replacing
their teachers with ours and removing validation-based checkpointing provided
in their code (as this is data-free KD where there is no ground truth
imagery). In the case of Tiny ImageNet, we did need to decrease the batch size
due to memory, however we proportionally increased the number of iterations
for each epoch to account for this. Besides adjusting the batch size in this
one situation, all hyperparameter settings are as specified in the respective
works.

Comparisons of CMI, PRE, and our method for CIFAR10, CIFAR100, and Tiny
ImageNet are shown in Table 3, with the best scores emphasized in bold. We
examined scenarios of self-distillation from teacher to a freshly initialized
teacher and distilling from teacher to a smaller, freshly initialized student.
We used WRN22x8, RX29-64x4, and RX29-64x8 as the teacher models for CIFAR10,
CIFAR100, and Tiny ImageNet, respectively, and RN18 as the student network for
all three datasets. Thus, our teacher networks are different from those used
in the original works for CMI and PRE, but the student network is exactly the
same. As mentioned, we do include the results for CMI and PRE reported in [4]
(including their reported baseline network scores) which used a
RN34$\rightarrow$RN18 network pair for all three datasets. Note that [4] also
observed difficulties employing CMI to Tiny ImageNet, thus they did not
provide any scores for that dataset.

As can be seen in Table 3, our approach significantly outperforms CMI and PRE
on all datasets and distillation scenarios for the values reported in [4] and
our obtained scores. It could be argued that we did not search for the optimal
hyperparameter values for the other approaches with respect to our teacher
networks. However, we reiterate that we used the settings provided by the
authors, indicating that substantial hyperparameter tuning is always required
when employing different teacher networks [17]. Our approach on the other hand
requires very minimal amounts of hyperparameter tuning and as will be shown in
the next set of experiments, our method is much more robust to the choice of
settings. Furthermore, the scores we obtained with our approach have
significantly less variability compared to CMI and PRE, as shown by the
standard deviation of the results.

|  | Accuracy
---|---|---
Dataset |  Method |  T |  S |  TA$\rightarrow$SA |  TA$\rightarrow$SB
C10 A: WRN22x8 B: RN18 |  CMI* |  95.40 |  95.20 |  - |  82.40$\pm$4.07
PRE* |  |  |  - |  87.40$\pm$3.21
CMI |  96.21 |  95.23 |  76.74$\pm$3.47 |  64.69$\pm$3.68
PRE |  |  |  87.69$\pm$1.30 |  60.67$\pm$16.7
Ours |  |  |  95.18$\pm$0.09 |  94.12$\pm$0.05
C100 A: RX29-64x4 B: RN18 |  CMI* |  77.90 |  77.1 |  - |  55.20$\pm$4.91
PRE* |  |  |  - |  70.20$\pm$3.33
CMI |  77.82 |  74.30 |  54.63$\pm$0.34 |  46.60$\pm$0.73
PRE |  |  |  56.41$\pm$3.62 |  41.39$\pm$4.44
Ours |  |  |  76.67$\pm$0.04 |  71.73$\pm$0.13
Tiny ImgNet A: RX29-64x8 B: RN18 |  CMI* |  71.20 |  64.90 |  - |  -
PRE* |  |  |  - |  46.30$\pm$3.32
CMI |  64.16 |  61.81 |  33.80$\pm$0.24 |  30.81$\pm$0.50
PRE |  |  |  9.14$\pm$3.75 |  8.59$\pm$1.52
Ours |  |  |  62.09$\pm$0.24 |  56.05$\pm$0.22

Table 3: Comparison of related DFKD approaches on CIFAR10, CIFAR100, and Tiny
ImageNet. Results on RN34$\rightarrow$RN18 obtained from [4] are denoted as
CMI* and PRE*.

Ablation & Sensitivity Studies. In this section, we conducted three different
studies: an ablation study on our adversarial attack, a sensitivity study on
the attack choices and settings, and a sensitivity study on the dataset and
training settings. For these experiments, we specifically focused on CIFAR10
with a WRN22x8 teacher and a RN18 student. Results of all three studies are
presented in Table 4.

For the attack ablation study, we observed that even when both adversarial
attacks (“border” and “deeper”) are completely removed (_i.e_., just
performing KD with the synthetic examples and data augmentation), information
can still be transferred from the teacher to the student in a data-free
manner. This suggests that employing in-domain versus out-of-domain data for
standard KD does not necessarily matter as long as the dataset uniformly
samples the teacher’s output labels. Thus, in the case of the out-of-domain
experiment in [3], it is likely that the base out-of-domain data predictions
in the teacher model were long-tail imbalanced, which did not sample the
decision space in the teacher sufficiently for performing KD. Furthermore, we
see that our approach requires both the first border attack and the second
deeper attack to reach the best performance possible. Additionally, the
inclusion of the “pre”-success examples in the first border attack results in
greater performance than using the “post”-success examples alone. However,
filtering the examples based on the conditions outlined in Sect. 3.3 does not
produce significantly different results from our full approach.

Study |  Experiment |  T $\rightarrow$ S
---|---|---
- | Full Approach |  94.12$\pm$0.05
Ablation |  ($-$) Both Adversarial Attacks |  93.07$\pm$0.27
|  ($-$) “Border” First Attack Examples |  93.38$\pm$0.15
|  ($-$) “Deeper” Second Attack Examples |  93.01$\pm$0.11
|  ($-$) “Pre”-Success Examples |  93.10$\pm$0.17
|  ($-$) Filter Conditions |  94.19$\pm$0.19
Attack Sensitivity | PGD-based Attack |  91.62$\pm$0.70
|  $\epsilon=6$ Initial Attack Epsilon |  94.15$\pm$0.17
|  $\epsilon=14$ Initial Attack Epsilon |  94.12$\pm$0.22
|  $\epsilon=4$ Second Attack Epsilon |  94.23$\pm$0.16
| No Softmax Threshold |  94.04$\pm$0.15
| 0.50 Softmax Threshold |  93.92$\pm$0.18
| 0.75 Softmax Threshold |  94.11$\pm$0.13
Dataset and Training Sensitivity | Minimal Data Augmentation w/ mixup |  93.57$\pm$0.19
| No Data Augmentation w/ Mixup |  93.03$\pm$0.14
| Standard Data Augmentation w/o Mixup |  92.92$\pm$1.08
| Minimal Data Augmentation w/o Mixup |  89.40$\pm$0.67
| No Data Augmentation w/o Mixup |  34.67$\pm$5.11
| Attack with Normal & Mixup |  94.32$\pm$0.31
| Distill with Normal, Mixup & Adversarial |  94.37$\pm$0.05
| 500 Examples per Class |  62.83$\pm$35.0
| 1000 Examples per Class |  90.93$\pm$0.60
| 5000 Examples per Class |  94.59$\pm$0.21
| 200 Distillation Epochs |  92.32$\pm$0.72
| 800 Distillation Epochs |  94.61$\pm$0.11
Table 4: Ablation and sensitivity studies on CIFAR10 with a WRN22x8 teacher
and RN18 student.

Next, we investigated the impact of our choices for our adversarial attacks.
We altered the first border attack to be a Projected Gradient Descent (PGD)
attack (_i.e_., a random uniform perturbation of $\epsilon=10$ was applied to
each example before the attack was conducted) and saw much worse results than
using BIM. We observed that the initial perturbation of PGD tends to push most
examples toward being initially classified as one of a few classes. Thus,
likely leading our attack to mostly identify the decision boundary between
these few labels and the targets. As for the values of $\epsilon$ and the
softmax threshold of the first border attack (the only hyperparameters for our
method), we see that our approach is fairly robust to the values for these
hyperparameters. In comparison to generator-based methods (Table 3), we see
that our approach is much less sensitive to settings.

To conclude our ablation and sensitivity studies, we examined the sensitivity
of our approach with respect to how much input data augmentation is utilized,
which examples are used in the adversarial attacks and in the distillation
loss, how many synthetic examples are used during distillation, and how long
the student network is trained. For the data augmentations, we experimented
with three different levels of data augmentations: standard (Sect. 4
Distillation Training Details), minimal (no RandAugment, random elastic
transform, or random inversion), and none. Additionally, we investigated the
effects of removing mixup from our approach. From Table 4, it is obvious to
see how crucial data augmentation and mixup are to our method. When no data
augmentation is applied, mixup alone enables our approach to achieve over 90%
accuracy on CIFAR10, which is about 60% higher than no data augmentation and
no mixup. Conversely, when standard data augmentation is used but mixup is
not, we see similar performance. Combining strong amounts of data augmentation
and mixup together helps achieve the best performance possible.

Next, we examined the effects of using more types of examples in the
adversarial attacks and in the distillation loss. By including the original
data augmented synthetic examples (pre-mixup) with the mixup examples in the
adversarial attacks, performance gains can be seen. Furthermore, when
including both the original data augmented synthetic examples and the mixup
examples (pre-attack) with the adversarially perturbed examples in the KD loss
computation, performance gains can also be seen. We also observed that sizable
performance gains can be obtained by increasing either the number of synthetic
examples per class or the number of epochs for performing KD. Through this
sensitivity study, we show that, while we obtained state-of-the-art results in
our previous experiments, there are several avenues for increasing performance
scores even further. Thus, in our CIFAR100 and Tiny ImageNet experiments and
in our “distilling down” experiments, it is likely that we could bridge the
gap between our distilled students and the baseline cross entropy student by
increasing the number of synthetic examples and conducting more distillation
epochs.

###  4.2 Privacy

Lastly, as recent works have shown that information about the training data
can be extracted from various models [7, 6, 24], we conducted a qualitative
privacy examination on our teacher models and distilled students in the
context of model inversion and data extraction. For this experiment, we
synthesized images using DeepInversion [64] applied to three MobileNetV2
models trained on CIFAR10. One is the teacher network that was trained with
the original dataset, one is a student network that was self-distilled from
the aforementioned teacher using the original CIFAR10 data, and the last is a
student network that was self-distilled from the aforementioned teacher using
our approach. As mentioned in [64], their approach for model inversion can be
sensitive to hyperparameters, thus we examined over 1000 different
hyperparameter configurations and selected examples from the configuration
yielding the lowest overall loss. These images are presented in Fig. 3, where
all images shown were predicted correctly by their respective model.

It is obvious to see that recognizable images of a horse, airplane, truck, and
bird were able to be extracted from the teacher model. Furthermore, we are
also able to obtain slightly less detailed, but still recognizable, images of
a horse, ship, truck, and bird from the CIFAR10 distilled student. However,
when we look at the examples obtained using our synthetic distilled student
model, we see that they are largely incoherent. In a couple of cases you can
see vague depictions of what could be a cat or a car, but immense detail is
missing. For the most part, images extracted from our distilled students
appeared to be abstract textures that contained no recognizable objects. Thus,
our approach could serve as a method for anonymizing the data that gets
naturally embedded into deep neural networks from standard supervised training
and ordinary KD.

Figure 3: Extracted CIFAR10 images from a MobileNetV2 teacher model (top row),
a CIFAR10-distilled MobileNetV2 student model (middle row), and a OpenGL-
distilled MobileNetV2 student model (bottom row).

##  5 Conclusion

We proposed a novel approach to data-free knowledge distillation that utilizes
procedurally rendered OpenGL shader images, combined with heavy data
augmentation and adversarial attacks, to transfer knowledge from a teacher
network to a new student model. Our work is straightforward to implement and
built upon standard KD, making it extendable to more advanced methods.
Experiments demonstrated improved and more stable results over relevant DFKD
approaches, establishing new state-of-the-art scores for multiple datasets.
Additionally, we showed that our technique can better anonymize the data
embedded in the teacher. While we presented results that are superior to other
methods, we also see opportunities for improvements (as shown in Table 4)
which we will explore in the future.

Acknowledgements. This research was supported by the U.S. Air Force Research
Laboratory under Contract #GRT00054740 (Release #AFRL-2023-5248). We would
additionally like to thank Skylar Wurster for his assistance.

## References

  * [1] A. Abusnaina, Y. Wu, S. Arora, Y. Wang, F. Wang, H. Yang, and D. Mohaisen.  Adversarial Example Detection using Latent Neighborhood Graph.  In IEEE/CVF International Conference on Computer Vision, 2021.
  * [2] M. Baradad, R. Chen, J. Wulff, T. Wang, R. Feris, A. Torralba, and P. Isola.  Procedural Image Programs for Representation Learning.  Advances in Neural Information Processing Systems, 2022.
  * [3] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov.  Knowledge Distillation: A Good Teacher is Patient and Consistent.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [4] K. Binici, S. Aggarwal, N. T. Pham, K. Leman, and T. Mitra.  Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay.  In AAAI Conference on Artificial Intelligence, 2022.
  * [5] C. Bucilua, R. Caruana, and A. Niculescu-Mizil.  Model Compression.  In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006.
  * [6] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramer, B. Balle, D. Ippolito, and E. Wallace.  Extracting Training Data from Diffusion Models.  arXiv:2301.13188, 2023.
  * [7] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al.  Extracting Training Data from Large Language Models.  In USENIX Security Symposium, 2021.
  * [8] N. Carlini and D. Wagner.  Towards Evaluating the Robustness of Neural Networks.  In IEEE Symposium on Security and Privacy, 2017.
  * [9] H. Chen, Y. Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian.  Data-Free Learning of Student Networks.  In IEEE/CVF International Conference on Computer Vision, 2019.
  * [10] J. H. Cho and B. Hariharan.  On the Efficacy of Knowledge Distillation.  In IEEE/CVF International Conference on Computer Vision, 2019.
  * [11] Y. Choi, J. Choi, M. El-Khamy, and J. Lee.  Data-Free Network Quantization with Adversarial Knowledge Distillation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
  * [12] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le.  RandAugment: Practical Automated Data Augmentation with a Reduced Search Space.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
  * [13] J. Davis and L. Frank.  Revisiting Batch Norm Initialization.  In European Conference on Computer Vision, 2022.
  * [14] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.  Scaling Vision Transformers to 22 Billion Parameters.  In International Conference on Machine Learning, 2023.
  * [15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In International Conference on Learning Representations, 2021.
  * [17] G. Fang, J. Song, X. Wang, C. Shen, X. Wang, and M. Song.  Contrastive Model Inversion for Data-Free Knolwedge Distillation GitHub.  https://github.com/zju-vipa/CMI.  Accessed: September 15, 2023.
  * [18] G. Fang, J. Song, X. Wang, C. Shen, X. Wang, and M. Song.  Contrastive Model Inversion for Data-Free Knolwedge Distillation.  In International Joint Conference on Artificial Intelligence, 2021.
  * [19] T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar.  Born Again Neural Networks.  In International Conference on Machine Learning, 2018.
  * [20] J. Geiping, M. Goldblum, G. Somepalli, R. Shwartz-Ziv, T. Goldstein, and A. Gordon-Wilson.  How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization.  In International Conference on Learning Representations, 2023.
  * [21] Y. Gong, L. Liu, M. Yang, and L. Bourdev.  Compressing Deep Convolutional Networks using Vector Quantization.  arXiv:1412.6115, 2014.
  * [22] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative Adversarial Nets.  In Advances in Neural Information Processing Systems, 2014.
  * [23] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.  On Calibration of Modern Neural Networks.  In International Conference on Machine Learning, 2017.
  * [24] N. Haim, G. Vardi, G. Yehudai, O. Shamir, and M. Irani.  Reconstructing Training Data from Trained Neural Networks.  Advances in Neural Information Processing Systems, 2022.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun.  Deep Residual Learning for Image Recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
  * [26] B. Heo, M. Lee, S. Yun, and J. Y. Choi.  Knowledge Distillation with Adversarial Samples Supporting Decision Boundary.  In AAAI Conference on Artificial Intelligence, 2019.
  * [27] G. Hinton, O. Vinyals, and J. Dean.  Distilling the Knowledge in a Neural Network.  arXiv Preprint arXiv:1503.02531, 2015.
  * [28] S. Ioffe and C. Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In International Conference on Machine Learning, 2015.
  * [29] S. A. Janowsky.  Pruning Versus Clipping in Neural Networks.  Physical Review A, 1989.
  * [30] D. Jones.  Good Practice in (Pseudo) Random Number Generation for Bioinformatics Applications.  Technical report, University College London, 2010.
  * [31] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  arXiv:1312.6114], 2013.
  * [32] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment Anything.  arXiv:2304.02643, 2023.
  * [33] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby.  Big Transfer (BiT): General Visual Representation Learning.  In European Conference on Computer Vision, 2020.
  * [34] A. Krizhevsky.  Learning Multiple Layers of Features from Tiny Images.  2009\.
  * [35] A. Kurakin, I. Goodfellow, and S. Bengio.  Adversarial Examples in the Physical World, 2016.
  * [36] J. Lust and A. P. Condurache.  GraN: An Efficient Gradient-Norm Based Detector for Adversarial and Misclassified examples.  In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2020.
  * [37] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun.  ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design.  In European Conference on Computer Vision, 2018.
  * [38] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In International Conference on Learning Representations, 2018.
  * [39] A. Mishra and D. Marr.  Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy.  In International Conference on Learning Representations, 2018.
  * [40] G. K. Nayak, K. R. Mopuri, V. Shaj, V. B. Radhakrishnan, and A. Chakraborty.  Zero-Shot Knowledge Distillation in Deep Networks.  In International Conference on Machine Learning, 2019.
  * [41] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami.  Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks.  In IEEE Symposium on Security and Privacy, 2016.
  * [42] M. Phuong and C. H. Lampert.  Distillation-Based Training for Multi-Exit Architectures.  In IEEE/CVF International Conference on Computer Vision, 2019.
  * [43] D. Picard.  torch.manual_seed(3407) Is All You Need: On the Influence of Random Seeds in Deep Learning Architectures for Computer Vision.  arXiv preprint arXiv:2109.08203, 2021.
  * [44] E. Raff, J. Sylvester, S. Forsyth, and M. McLean.  Barrage of Random Transforms for Adversarially Robust Defense.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [45] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio.  FitNets: Hints for Thin Deep Nets.  In International Conference on Learning Representations, 2015.
  * [46] S. Zagoruyko and N. Komodakis.  Wide Residual Networks.  In British Machine Vision Conference, 2016.
  * [47] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  MobileNetV2: Inverted Residuals and Linear Bottlenecks.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  * [48] D. Saxena and J. Cao.  Generative Adversarial Networks (GANs) Challenges, Solutions, and Future Directions.  ACM Computing Surveys, 2021.
  * [49] Z. Shen and M. Savvides.  MEAL v2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet Without Tricks.  In Advances in Neural Information Processing Systems Workshops, 2020.
  * [50] D. Shreiner et al.  OpenGL Programming Guide: The Official Guide to Learning OpenGL, Versions 3.0 and 3.1.  Pearson Education, 2009.
  * [51] R. Singh and S. S. Gill.  Edge AI: A Survey.  Internet of Things and Cyber-Physical Systems, 2023.
  * [52] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.  Intriguing Properties of Neural Networks.  arXiv:1312.6199, 2013.
  * [53] A. Tarvainen and H. Valpola.  Mean Teachers are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results.  Advances in Neural Information Processing Systems, 2017.
  * [54] Y. Tian, D. Krishnan, and P. Isola.  Contrastive Representation Distillation.  In International Conference on Learning Representations, 2020.
  * [55] Z. Tian, Z. Wang, A. M. Abdelmoniem, G. Liu, and C. Wang.  Knowledge Representation of Training Data with Adversarial Examples Supporting Decision Boundary.  IEEE Transactions on Information Forensics and Security, 2023.
  * [56] TwiGL.  TwiGL.  https://github.com/doxas/twigl.  Accessed: June 29, 2023.
  * [57] G. Urban, K. J. Geras, S. E. Kahou, O. Aslan, S. Wang, A. Mohamed, M. Philipose, M. Richardson, and R. Caruana.  Do Deep Convolutional Nets Really Need to be Deep and Convolutional?  In International Conference on Learning Representations, 2017.
  * [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, Ł. Kaiser, and I. Polosukhin.  Attention Is All You Need.  In Advances in Neural Information Processing Systems, 2017.
  * [59] Y. Wang, J. Wang, W. Zhang, Y. Zhan, S. Guo, Q. Zheng, and X. Wang.  A Survey on Deploying Mobile Deep Learning Applications: A Systemic and Technical Perspective.  Digital Communications and Networks, 2022.
  * [60] Z. Wang.  Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis.  In AAAI Conference on Artificial Intelligence, 2021.
  * [61] C. Xie, Y. Wu, L. van der Maaten, A. L. Yuille, and K. He.  Feature Denoising for Improving Adversarial Robustness.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [62] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le.  Self-Training with Noisy Student Improves ImageNet Classification.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [63] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated Residual Transformations for Deep Neural Networks.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
  * [64] H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz.  Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [65] J. Yoo, M. Cho, T. Kim, and U. Kang.  Knowledge Extraction with No Observable Data.  Advances in Neural Information Processing Systems, 2019.
  * [66] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  MixUp: Beyond Empirical Risk Minimization.  In International Conference on Learning Representations, 2018.


11institutetext: Kensho Technologies, Cambridge, MA, 02138, USA
11email: {jilin.wang, michael.krumdick, hamima.halim, maxim.sokolov,
vadym<EMAIL_ADDRESS>22institutetext: Meta, Cambridge, MA, 02140,
USA
22email<EMAIL_ADDRESS>33institutetext: Google, Los Angeles, CA, 90291,
USA
33email<EMAIL_ADDRESS>
# A Graphical Approach to Document Layout Analysis

Jilin Wang 11** Michael Krumdick 11** Baojia Tong 22 Hamima Halim 11 Maxim
Sokolov 11 Vadym Barda 11 Delphine Vendryes 33 Chris Tanner 11

###### Abstract

Document layout analysis (DLA) is the task of detecting the distinct, semantic
content within a document and correctly classifying these items into an
appropriate category (e.g., text, title, figure). DLA pipelines enable users
to convert documents into structured machine-readable formats that can then be
used for many useful downstream tasks. Most existing state-of-the-art (SOTA)
DLA models represent documents as images, discarding the rich metadata
available in electronically generated PDFs. Directly leveraging this metadata,
we represent each PDF page as a structured graph and frame the DLA problem as
a graph segmentation and classification problem. We introduce the Graph-based
Layout Analysis Model (GLAM), a lightweight graph neural network competitive
with SOTA models on two challenging DLA datasets - while being an order of
magnitude smaller than existing models. In particular, the 4-million parameter
GLAM model outperforms the leading 140M+ parameter computer vision-based model
on 5 of the 11 classes on the DocLayNet dataset. A simple ensemble of these
two models achieves a new state-of-the-art on DocLayNet, increasing mAP from
76.8 to 80.8. Overall, GLAM is over 5 times more efficient than SOTA models,
making GLAM a favorable engineering choice for DLA tasks.

11footnotetext: The authors contributed equally to this work.

##  1 Introduction

Much of the world’s information is stored in PDF (Portable Document Format)
documents; according to data collected by the Common
Crawl***https://commoncrawl.github.io/cc-crawl-statistics/plots/mimetypes, PDF
documents are the most-popular media type on the web other than HTML.

A standard PDF file contains a complete specification of the rendering of the
document. All objects (e.g., texts, lines, figures) in the PDF file include
their associated visual information, such as font information and location on
the page. However, a PDF file does not always contain information about the
relationships between those objects, which is critical for understanding the
structure of the document. For example, all text is stored at the individual
character level. Sentences from the characters and consequently paragraphs or
titles formed by the sentences are not included in the metadata. Therefore,
despite PDF files containing explicit, structured representations of their
content, correctly structuring this content into human-interpretable
categories remains a challenging problem and is the crux of Document Layout
Analysis (DLA).

Existing approaches to PDF extraction typically consist of two main components
— a PDF parser and DLA model — as depicted in Fig. 1 [32, 6]:

  1. 1.

PDF parser (e.g., PDF Miner): extracts the document’s underlying raw data,
often doing so by producing fine-grained, pertinent contiguous blocks of
information referred to as “objects” (e.g., a line of text). For a standard
(non-scanned) PDF document, the PDF parser is able to generate information
about each “object”: its location on the page, the raw content contained
within it, and some metadata like font and color about how the object is
displayed.

  2. 2.

DLA model: aims to group together objects that concern semantically similar
content (e.g., a table, figure, paragraph of text), essentially turning an
unordered collection of data into a coherent, structured document. The output
is a collection of bounding boxes, each representing one of the semantic
objects (e.g., a paragraph).

Figure 1: Visualization of the data extraction pipeline. We use pdfminer.six
as the PDF parser in our pipeline.

DLA models frequently take a computer vision-based approach [15, 21, 30].
These approaches rely on treating documents as images and ignoring much of the
rich, underlying metadata.

Generally, PDF extraction can be evaluated by the quality of the DLA model’s
segmentation predictions. The standard approach to evaluating the segmentation
performance is to use the mean average precision (mAP) metric to assess the
overlap between the predicted and ground-truth bounding boxes. Although this
is standard practice, mAP is sensitive to the bounding boxes’ exact pixel
values. Therefore, bounding box-level metrics are only proxies for measuring
segmentation performance, and ultimately a user wishes to measure the
performance of clustering objects to semantic segments. In this work, we
approach the problem by modeling the document not as a collection of pixels
but as a structured graphical representation of the output from a PDF parser.

Specifically, we frame the DLA task as a joint graph segmentation and node
classification problem. Using this formulation, we develop and evaluate a
graph neural network (GNN) based model – the Graph-based Layout Analysis Model
(GLAM). Our model is able to achieve performance that is comparable in mAP to
the current SOTA methods while being lightweight, easier to train, and faster
at inference time.

Our primary contributions are :

  1. 1.

Propose a novel formulation of DLA, whereby we create a structured graphical
representation of the parsed PDF data and treat it as a joint node
classification problem and graph segmentation problem.

  2. 2.

Develop a novel Graph-based Layout Analysis Model (GLAM) for this problem
formulation and evaluate it on two challenging DLA tasks: PubLayNet and
DocLayNet.

  3. 3.

Demonstrate that by ensembling our model with an object detection model, we
are able to set a new state-of-the-art (SOTA) on DocLayNet, increasing the mAP
on this task from 76.8 to 80.8.

  4. 4.

Validate that GLAM is compact in size and fast at speed with a comparison
against SOTA models, making it efficient and applicable in industrial use
cases.

##  2 Related Work

###  2.1 Document Layout Analysis

As a field, document layout analysis (DLA) is mainly aimed toward enterprise
use cases [3]. Businesses often need to process massive streams of documents
to complete complex downstream tasks, such as information retrieval, table
extraction, key-value extraction, etc. Early literature in this space focused
on parsing-based solutions that would attempt to understand the document
layout from bounding boxes and textual data parsed from born-digital files,
leveraging features from the native file type [6].

###  2.2 Object Detection-Based Methods in VRDU

Recent work has focused on Visually Rich Document Understanding (VRDU), which
models documents as images. Image-based object detection formulations of this
problem have been particularly favorable and popular because of their
generality; effectively any type of human-readable document can be converted
into an image. Further, using a traditional, simple convolutional neural
network (CNN) for feature extraction works well in generating representations
of documents. Early iterations of LayoutLM and others [31, 11] all adopt this
method.

Optionally, textual information can be captured by an optical character
recognition (OCR) engine, avoiding the need to integrate text parsing engines
for multiple document media types [8]. This OCR step, whose accuracy is
critical for any model that uses textual information [20], could be fine-tuned
on specific formats but will also introduce an upper bound of OCR accuracy.

Recently, there has been an emphasis on using large pre-trained document
understanding models that leverage vast, unlabeled corpora, such as the IIT-
CDIP test collection [19]. The Document Image Transformer (DiT) [21] model
adopts a BERT-like pre-training step to train a visual backbone, which is
further adapted to downstream tasks, including DLA. There have been additional
efforts in creating pre-trained multi-modal models that leverage both the
underlying document data and the visual representation. The LayoutLM series of
models [30, 16] have been particularly successful. LayoutLMv3 [15] is pre-
trained with Masked Language Modeling (MLM), Masked Image Modeling (MIM), and
Word-Patch Alignment (WPA) training objectives. The LayoutLMv3 is integrated
as a feature backbone in a Cascade R-CNN detector [5] with FPN [23] for
downstream DLA tasks.

###  2.3 Graph Neural Networks in Document Understanding

Graphs and graphical representations have been used extensively in document
understanding (DU) [4], especially toward the problem of table understanding.
[17] extracted saliency maps using a CNN and then used a Conditional Random
Field (CRF) model to reason about the relationships of the identified salient
regions for table and chart detection. Holeček et al. [14] used the parsed
data from a PDF to create a graphical representation of a document and then a
GNN for table detection and extraction. Qasim et al. [26] uses a computer-
vision feature augmented graph and a GNN for segmentation in order to perform
table structure recognition.

Graphs have been used in DLA as well. Zhang et al. [33] use data from a PDF
parser and a module based on graph neural networks to model the relationship
of potential candidate layout outputs as part of their larger, multi-modal
fusion architecture. A key component of any graph-based method is the
formulation of the graphs. While the edges have diverse interpretations, the
nodes will typically represent some logically contiguous region of the
document. In image-based methods, this can be obtained by computer vision, as
in [22], or by a more modular OCR task as leveraged by [24]. In contrast, Wei
et al. [29] and this work parse the ground truth text boxes if focusing on
digital-generated documents.

Typically, two nodes are connected by an edge if they satisfy certain spatial
criteria (e.g., based on their vertical or horizontal locations). Riba et al.
[27] notes that the resulting graph structures are similar to visibility
graphs, which represent nodes as geometric blocks of space that are connected
only if they have some vertical or horizontal overlap with no other nodes in
between.

Most similar to our work are [22] and [10]. Li et al. [22] also frames DLA as
a graph segmentation and node classification problem but uses computer vision
in order to build its “primitive” nodes rather than using PDF parsing. Gemelli
et al. [10] uses a GNN on a representation of the document similar to this
work. However, they focus only on node classification for table extraction.
Our model is able to perform the entire DLA task in an end-to-end fashion by
modeling both node-level classifications and graph segmentation.

##  3 Methodology

###  3.1 Graph Generation

Figure 2: Example document graph from PubLayNet with associated node and edge
labels. Left: text boxes parsed from the PDF, whereby categories are labeled
in different colors (blue for title, orange for text, and magenta for table).
Right: corresponding graph generated. Each node represents a text box, and the
arrows between nodes represent the edges. Illustratively, green edges indicate
a link (positive), and grey edges represent there is no link (negative).

We use a PDF parser to extract all text boxes, each of which also includes 79
associated features, such as bounding box location, text length, the fraction
of numerical characters, font type, font size, and other simple heuristics.
Using this parsed metadata, we construct a vector representation for each
individual box, which constitutes a node in our graph.

Bidirectional edges are established between nodes by finding each node’s
nearest neighbor in each possible direction (up, down, left, and right).
Additional edges are added to approximate the potential reading order of the
underlying text boxes in the document (i.e. the heuristic order parsed from
PDF documents – up to down, left to right). Each edge also has its own vector
representation, which includes the edge direction and the distance between the
two nodes that it connects. A visualization of the resulting graph can be seen
in Fig. 2.

###  3.2 Graph-based Layout Analysis Model

There are two key parts to the Graph-based Layout Analysis Model (GLAM): a
graph network and a computer vision feature extractor.

####  3.2.1 Graph Network

Figure 3: Visualization of the GNN architecture used in GLAM.

Our graph network model consists of a graph convolutional network (GCN) [18]
and two classification heads – one for node classification and one for edge
classification. The model structure is shown in Fig. 3. We apply batch
normalization to the node features before inputting them into a series of
linear and topology adaptive graph (TAG) convolutional [9] layers. We set the
TAG’s first hidden layer to have 512 dimensions when training on PubLayNet,
and we increase this to 1024 dimensions for the more complex DocLayNet
dataset. As the GCN network gets deeper, we scale down the number of
parameters in both linear and TAG convolutional layers by a factor of 2. We
concatenate this output with the original normalized features and then apply a
series of linear layers to create our node embeddings. A classification head
then projects this into a distribution over the possible segment classes
(e.g., title, figure, text). This model is very lightweight, totaling just
over one million parameters.

To create the edge embeddings, we extract the relevant node embeddings and
concatenate them with the original edge features. We batch-normalize this
representation and then apply another two linear layers to generate the final
edge embeddings.

The model is trained with a weighted, joint loss function Eq. 1 from the two
classification tasks.

$L=L_{node}+\alpha L_{edge}$ (1)

where $L_{node}$ and $L_{edge}$ are a cross-entropy loss for node and edge
classification, and the edge loss scale $\alpha=4$ is applied to improve edge
classification results for better segmentation.

Figure 4: Visualization of the model pipeline.

At inference time, segment clustering follows the steps shown in Fig. 4. For a
given input graph, the model will classify each node into one of the given
segment categories and predict whether each edge is positive or negative. The
negative edges are removed from the graph. The segments are then defined by
the remaining connected components in the graph. Each segment is classified by
the majority vote of the classification of all of its component nodes. This is
finally converted back into a COCO-style annotation by using the minimum
spanning box of the nodes within the segment as the bounding box, the selected
class label as the label, and the mean probability of that label over all of
the nodes as the score.

####  3.2.2 Computer Vision Feature Extraction

One of the main limitations of PDF-parser-based data pre-processing is that it
is unable to capture all of the visual information encoded in the PDF. All of
the text boxes are only represented by their metadata, which does not account
for many of the semantic visual cues within a layout. These cues include
background colors, lines, dots, and other non-textual document elements. To
include this visual information, we extend our node-level features with a set
of visual features. We render the PDF as an image and extract a visual feature
map using ResNet-18 [12]. Specifically, to extract the image features
corresponding to each input node’s bounding box, we use Region of Interest
(ROI) extraction with average pooling [13].

We concatenate these image features with our metadata-based feature vectors,
then apply a simple three-layer attention-based encoder model [28] to these
features to give them global context before concatenating them with the
features associated with the nodes in our graph. This maintains the overall
graphical structure while adding the ability to leverage additional visual
cues. The network is also relatively small, totaling only slightly over three
million parameters. This feature extraction process does not create any
additional nodes. Thus, while the model may be able to include some
information from image-rich regions of the document — such as figures and
pictures — the model is still limited in its ability to account for them.

##  4 Experiments

###  4.1 Datasets

Figure 5: Example document pages from PubLayNet (left) and DocLayNet (right)
datasets

PubLayNet [34] is a dataset consisting of 358,353 biomedical research articles
in PDF format (approximately 335k for training, 11k for development, and 11k
for testing). Each document is annotated with bounding boxes and labels for
every distinct semantic segment of content (i.e., text, title, list, figure,
and table). These bounding boxes are automatically generated by matching the
PDF format and the XML format of the articles. In practice, this leads to
bounding boxes that are often not perfectly aligned with the underlying
document. [22] finds that this process severely degrades the performance of
any method that generates boxes using data directly from the underlying
document. For an equal comparison across baselines, we maintain these boxes
without adjustment and report our raw mAP (mean average precision) score.

In this work, we primarily focused on using the DocLayNet [25] dataset, due to
it having a diverse set of unique layouts. While PubLayNet only contains
consistently formatted journal articles, DocLayNet spans six distinct genres
(i.e., financial reports, manuals, scientific articles, laws & regulations,
patents, and government tenders). It provides layout annotation bounding boxes
for 11 different categories (caption, footnote, formula, list-item, page
footer, page header, picture, section header, table, text, and title) on
80,863 unique document pages. The documents are divided into a training set
(69,375 documents), a development set (6,489 documents), and a testing set
(4,999 documents).

Unlike PubLayNet, DocLayNet is a human-annotated dataset; the ground truth
bounding boxes are generated by human annotators hand-drawing boxes on top of
the documents. These human-drawn boxes then are adjusted to match the
underlying extracted text boxes, removing any additional whitespace or
annotator errors in the process. This ensures a perfect correlation between
the bounding boxes defined by the extracted data and the ground truth labels.
For classes that are not well aligned with the underlying text boxes (e.g.,
picture and table), the boxes are left unmodified. The current state-of-the-
art DLA model for this dataset is an off-the-shelf YOLO v5x6 [2] model.

###  4.2 Data Pre-processing

In conventional VRDU-based methods, PDFs are first converted into images
before inference. However, many ground truth features — not only the raw text
but text and artifact formatting — can be lost in the image conversion step if
the original file information is discarded. Thus, we parse text boxes directly
from the PDF documents, which allows us to acquire precise text content,
position, and font information.

When using the DocLayNet dataset, we directly use the provided parsed text
boxes, which include the bounding box location, the text within the box, and
information about its font. These boxes are only provided for textual content,
as opposed to figures, tables, etc.

When using the PubLayNet dataset, text boxes are parsed directly from original
PDF pages using an in-house PDF parser based on pdfminer.six [1]. To better
match DocLayNet, we ignore image-based objects in the document entirely. We
apply a series of data cleaning steps (e.g. ignore random symbols in some
documents by removing very small bounding boxes smaller than 10 pixels in
width and height along with extra white space) on the parsed boxes. To reduce
the number of unnecessary edge classifications between adjacent nodes, we use
simple distance-based heuristics to merge adjacent boxes. This simplifies the
graph structure and reduces the overall memory requirements during training
and inference.

Both DocLayNet and PubLayNet contain ground truth data in the COCO format. In
order to convert this format to labels on our graph, we take the bounding box
for each annotation and find all of the text boxes in the document that have
some overlap. We compute the minimum spanning box of this set of text boxes
and compare it with the original ground truth. For DocLayNet, if the IoU of
the ground truth box and the box formed by all the overlapping text boxes is
less than 0.95, we remove text boxes in order of their total area outside the
annotation box and select the configuration that achieves the highest IoU.

Once we have associated every annotation with its set of text boxes and their
corresponding nodes on the graph, we label each node on the graph with the
class of the annotation. Every edge that connects two nodes within the same
annotation is labeled a positive edge, while every edge connecting nodes from
different annotations is labeled as a negative edge. An example of the labeled
graph is presented in Fig. 2.

Table 1: Model performance mAP@IoU[0.5:0.95] on the DocLayNet test set, as compared to metrics reported in [25]. |  MRCNN R50 |  MRCNN R101 |  FRCNN R101 |  YOLO v5x6 |  GLAM |  GLAM + YOLO v5x6
---|---|---|---|---|---|---
caption |  68.4 |  71.5 |  70.1 |  77.7 |  74.3 |  77.7
footnote |  70.9 |  71.8 |  73.7 |  77.2 |  72.0 |  77.2
formula |  60.1 |  63.4 |  63.5 |  66.2 |  66.6 |  66.6
list item |  81.2 |  80.8 |  81.0 |  86.2 |  76.7 |  86.2
page footer |  61.6 |  59.3 |  58.9 |  61.1 |  86.2 |  86.2
page header |  71.9 |  70.0 |  72.0 |  67.9 |  78.0 |  78.0
picture |  71.7 |  72.7 |  72.0 |  77.1 |  5.7 |  77.1
section header |  67.6 |  69.3 |  68.4 |  74.6 |  79.8 |  79.8
table |  82.2 |  82.9 |  82.2 |  86.3 |  56.3 |  86.3
text |  84.6 |  85.8 |  85.4 |  88.1 |  74.3 |  88.1
title |  76.6 |  80.4 |  79.9 |  82.7 |  85.1 |  85.1
overall |  72.4 |  73.5 |  73.4 |  76.8 |  68.6 |  80.8

###  4.3 Results

####  4.3.1 DocLayNet

As shown in Table 1, GLAM achieves competitive performance while being over
10$\times$ smaller than the smallest comparable model — and over 35$\times$
smaller than the current state-of-the-art. Our model outperforms all of the
object detection models on five of the 11 total classes, yielding an overall
mAP of 68.6. The SOTA model achieves a 76.8.

Table 2: Model performance mAP@IoU[0.5:0.95] on the PubLayNet val set. |  LayoutLM v3 [15] |  MBC [22] |  VSR [33] |  YOLO v5x6 |  GLAM
---|---|---|---|---|---
text |  94.5 |  88.8 |  96.7 |  91.7 |  87.8
title |  90.6 |  52.8 |  93.1 |  79.1 |  80.0
list item |  95.5 |  88.1 |  94.7 |  86.9 |  86.2
table |  97.9 |  97.7 |  97.4 |  96.5 |  86.8
figure |  97.0 |  84.0 |  96.4 |  94.7 |  20.6
overall |  95.1 |  82.3 |  95.7 |  89.8 |  72.2

Nine of the 11 classes have annotations that involve text box snapping; the
tables and pictures classes do not. When considering only these nine classes,
our model achieves an mAP score of 77.0, compared to 75.7 for the next-best
model. As expected, our model does not perform as well for tables, and it is
particularly low-performing for pictures. When only considering the parsed
text of tables (and thus disregarding visual markers such as table borders),
our model yields a low mAP for the table class. In fact, GLAM performs well in
extracting the compact regions of text from table contents (Fig. 6). If we
consider all 10 text-based classes (including tables but excluding pictures),
GLAM achieves an mAP score of 74.9, compared to 76.8 from the SOTA model. By
creating a simple ensemble with GLAM and YOLO v5x6, we are able to increase
the state of the art from 76.8 to 80.8.

Figure 6: Examples of GLAM predictions on PubLayNet documents. Bounding boxes
in orange, blue, green, magenta, and red colors represent predicted segments
in text, title, list, table, and figure categories, respectively. Due to the
inability to parse images from PDF, the GLAM model underperforms in figure
categories.

####  4.3.2 PubLayNet

The results on PubLayNet are shown in Table 2. On this dataset, our model is
significantly handicapped by the misalignment between the text boxes and the
ground truth boxes as shown in Figure 7. Since it cannot improve the alignment
of the original text boxes, this enacts a hard upper bound on performance. The
model can only achieve an IoU as high as the overlap between the ground-truth
label and the underlying PDF data. Although our method may correctly identify
and label all of the underlying nodes, it may still not achieve an IoU high
enough to register as a correct classification at most of the thresholds.

We can get a more accurate picture of performance by looking at the lower IoU
thresholds. At a threshold of 0.5, GLAM achieves an mAP higher than 90 for all
of the text classes. For the “title” class, the gap between mAP at this
threshold and the final value averaged over the full threshold range is over
17%. As a comparison, this gap on DocLayNet is less than 7% for every class.
Even with this misalignment, the GLAM model still is able to achieve 88% of
the performance of the SOTA model on the text-based categories.

Figure 7: Examples of bounding box misalignment between GLAM prediction and
annotation in PubLayNet. Red bounding boxes are from annotation, and blue
bounding boxes are from GLAM prediction. GLAM predictions have tight bounding
boxes around text-based segments.

####  4.3.3 Analysis

In general, our model performs strongest on small, text-box-based classes,
such as titles and headers. We hypothesize that this is mostly due to its
ability to perfectly match its bounding box to the ground truth label. Since
GLAM has access to the exact text boxes in question, it can output pixel-
perfect annotations by simply segmenting the relevant boxes. Object detection
models on the other hand are known to struggle with smaller objects [7].

Conversely, our model performs worst on classes that are not well captured by
the underlying text boxes in the document. This is a fundamental limitation of
our structural representation of the document. This discussion is continued in
Section 5.

###  4.4 CV Feature Ablation

Table 3: CV Feature extraction Ablation, mAP@IoU[0.5:0.95] on DocLayNet test set. | GLAM | GLAM-CV | $\Delta$
---|---|---|---
caption | 74.3 | 60.6 | 13.7
footnote | 72.0 | 61.1 | 10.9
formula | 66.6 | 43.8 | 22.8
list item | 76.7 | 61.5 | 15.2
page footer | 86.2 | 87.3 | -1.1
page header | 78.0 | 78.1 | -0.1
picture | 5.7 | 4.3 | 1.4
section header | 79.8 | 70.1 | 9.7
table | 56.3 | 50.2 | 6.1
text | 74.3 | 68.4 | 5.9
title | 85.1 | 72.9 | 12.2
overall | 68.6 | 59.9 | 8.79
Table 4: CV Feature extraction Ablation, mAP@IoU[0.5:0.95] on PubLayNet val set. | GLAM | GLAM - CV | $\Delta$
---|---|---|---
text | 87.8 | 85.9 | 1.9
title | 80.0 | 79.8 | 0.2
list item | 86.2 | 84.8 | 1.4
table | 86.8 | 86.1 | 0.7
figure | 20.6 | 18.3 | 2.3
overall | 72.2 | 71.0 | 1.2

We include an ablation study of the computer vision feature extractor in
Tables 3 and 4. For these models, we only train the graph network component of
GLAM, without the CV features. The GNN component represents only one-fourth of
the total parameters of the GLAM. Removing the vision feature extraction leads
to a consistent decrease in performance across both tasks, albeit only
slightly on PubLayNet. This is likely the result of two factors: (1) our model
nearing the performance limit imposed by the misaligned boxes in PubLayNet;
and (2) the visual complexity of the DocLayNet data. DocLayNet layouts contain
many visual cues that are not captured by the underlying text boxes (e.g.,
bullet points, lines, and colored backgrounds). Results on the PubLayNet
dataset (Table 4) indicate that the GNN component in GLAM already performs
well on the visual-simple type of documents like research articles.

###  4.5 Model Size and Efficiency

Table 5: Comparison of model size and efficiency. All numbers are bench-marked on a G4 AWS instance, with NVIDIA Tesla T4 GPU. | Model Size | Inference Time (ms) | Pages/sec
---|---|---|---
| (M) | GPU | CPU | GPU | CPU
LayoutLMv3 | 133 | 687 | 6100 | 1.5 | 0.2
YOLO v5 | 140.7 | 56 | 179 | 17.8 | 5.6
GLAM | 4 | 10 | 16 | 98.0 | 63.7
GLAM - CV | 1 | 4 | 9 | 243.9 | 109.9

Although large, pre-trained models often lead to high performance when adapted
to downstream tasks, they tend to be relatively large and slow in training and
inference. Both the state-of-the-art object detection models on PubLayNet and
DocLayNet have over 130 million parameters. To highlight the efficiency of our
graph-based model, in Table 5 we include additional benchmarks regarding
inference speed and throughput. We baseline our model against the two highest-
performing, open-source models on each task.

In general, GLAM is much more efficient than the baseline models. The full
GLAM model, which achieves competitive performance to the baseline models, is
able to process a document page in 10 ms on a single GPU†††We use an NVIDIA
Tesla T4 as GPU in our experiments., compared to 56 ms and 687 ms for YOLO
v5x6 and LayoutLMv3, respectively. Removing the CV feature extraction drops
the inference time to 4 ms, allowing the model to process over 243 document
pages per second on a single GPU. Such efficient DLA models significantly
reduce the engineering complexity in real-world use cases and enable the
possibility of on-device model deployments.

##  5 Limitations

The main limitation of GLAM is that it is highly dependent on the parsing
quality of the document. The benefit of using off-the-shelf object detection
models is that their performance is much less sensitive to the structure of
the underlying document. That is, they can account for the misaligned ground-
truth boxes, such as the ones in PubLayNet. Importantly, they can also
properly work with scanned documents and other documents with little or no
structure – alleviating the need to address pre-processing decisions. For the
same reason, GLAM is unable to capture many regions of vision-rich segments
(e.g., figures and pictures) in PDF documents. GLAM also does not take into
account any of the available document semantic information. To address this
limitation, we could use an additional text feature extractor (text embedding)
for the text boxes.

##  6 Conclusion

In this work, we introduce GLAM, a lightweight graph-network-based method for
performing document layout analysis on PDF documents. We build upon a novel,
structured representation of the document created from directly parsing the
PDF data. We frame the document layout analysis task as performing node and
edge classification on document graphs. With these insights, GLAM performs
comparably to competitive baseline models while being orders of magnitude
smaller. The model works particularly well for text-based layout elements such
as headers and titles. By ensembling this model with an off-the-shelf object
detection model, we are able to set state-of-the-art performance in DocLayNet.
We demonstrate that GLAM is a highly-efficient and effective approach to
extracting and classifying text from PDF documents, which makes it
competitive, especially in real-world applications with limited computational
resources.

## References

  * pdf [2022] pdfminer.six.  https://github.com/pdfminer/pdfminer.six, 2022.
  * yol [2022] Yolov5 sota realtime instance segmentation.  https://github.com/ultralytics/yolov5, 2022.
  * Binmakhashen and Mahmoud [2019] Galal M Binmakhashen and Sabri A Mahmoud.  Document layout analysis: a comprehensive survey.  _ACM Computing Surveys (CSUR)_ , 52(6):1–36, 2019\.
  * Bunke and Riesen [2011] Horst Bunke and Kaspar Riesen.  Recent advances in graph-based pattern recognition with applications in document analysis.  _Pattern Recogn._ , 44(5):1057–1067, may 2011\.  ISSN 0031-3203.  doi: 10.1016/j.patcog.2010.11.015.  URL https://doi.org/10.1016/j.patcog.2010.11.015.
  * Cai and Vasconcelos [2018] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6154–6162, 2018.
  * Chao and Fan [2004] Hui Chao and Jian Fan.  Layout and content extraction for pdf documents.  In Simone Marinai and Andreas R. Dengel, editors, _Document Analysis Systems VI_ , pages 213–224, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg.  ISBN 978-3-540-28640-0.
  * Chen et al. [2020] Guang Chen, Haitao Wang, Kai Chen, Zhijun Li, Zida Song, Yinlong Liu, Wenkai Chen, and Alois Knoll.  A survey of the four pillars for small object detection: Multiscale representation, contextual information, super-resolution, and region proposal.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , PP:1–18, 07 2020.  doi: 10.1109/TSMC.2020.3005231.
  * Cui et al. [2021] Lei Cui, Yiheng Xu, Tengchao Lv, and Furu Wei.  Document ai: Benchmarks, models and applications.  _arXiv preprint arXiv:2111.08609_ , 2021.
  * Du et al. [2017] Jian Du, Shanghang Zhang, Guanhang Wu, José MF Moura, and Soummya Kar.  Topology adaptive graph convolutional networks.  _arXiv preprint arXiv:1710.10370_ , 2017.
  * Gemelli et al. [2022] Andrea Gemelli, Emanuele Vivoli, and Simone Marinai.  Graph neural networks and representation embedding for table extraction in pdf documents.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , pages 1719–1726, 2022.  doi: 10.1109/ICPR56361.2022.9956590.
  * Gu et al. [2022] Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Nikolaos Barmpalios, Rajiv Jain, Ani Nenkova, and Tong Sun.  Unified pretraining framework for document understanding.  _arXiv e-prints_ , pages arXiv–2204, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.  doi: 10.1109/CVPR.2016.90.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2980–2988, 2017.  doi: 10.1109/ICCV.2017.322.
  * Holeček et al. [2019] Martin Holeček, Antonín Hoskovec, Petr Baudiš, and Pavel Klinger.  Table understanding in structured documents.  In _2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)_ , volume 5, pages 158–164, 2019.  doi: 10.1109/ICDARW.2019.40098.
  * Huang et al. [2022a] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  _arXiv preprint arXiv:2204.08387_ , 2022a.
  * Huang et al. [2022b] Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  In _ACM Multimedia 2022_ , October 2022b.  URL https://www.microsoft.com/en-us/research/publication/layoutlmv3-pre-training-for-document-ai-with-unified-text-and-image-masking/.
  * Kavasidis et al. [2019] I. Kavasidis, C. Pino, S. Palazzo, F. Rundo, D. Giordano, P. Messina, and C. Spampinato.  A saliency-based convolutional neural network for table and chart detection in digitized documents.  In Elisa Ricci, Samuel Rota Bulò, Cees Snoek, Oswald Lanz, Stefano Messelodi, and Nicu Sebe, editors, _Image Analysis and Processing – ICIAP 2019_ , pages 292–302, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-30645-8.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=SJU4ayYgl.
  * Lewis et al. [2006] D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard.  Building a test collection for complex document information processing.  In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’06, page 665–666, New York, NY, USA, 2006. Association for Computing Machinery.  ISBN 1595933697.  doi: 10.1145/1148170.1148307.  URL https://doi.org/10.1145/1148170.1148307.
  * Li et al. [2021] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si.  StructuralLM: Structural pre-training for form understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6309–6318, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.493.  URL https://aclanthology.org/2021.acl-long.493.
  * Li et al. [2022] Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei.  Dit: Self-supervised pre-training for document image transformer.  _arXiv preprint arXiv:2203.02378_ , 2022.
  * Li et al. [2020] Xiao-Hui Li, Fei Yin, and Cheng-Lin Liu.  Page segmentation using convolutional neural network and graphical model.  In Xiang Bai, Dimosthenis Karatzas, and Daniel Lopresti, editors, _Document Analysis Systems_ , pages 231–245, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-57058-3.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Liu et al. [2019] Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao.  Graph convolution for multimodal information extraction from visually rich documents.  In _NAACL_ , 2019.
  * Pfitzmann et al. [2022] Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter WJ Staar.  Doclaynet: A large human-annotated dataset for document-layout analysis.  _arXiv preprint arXiv:2206.01062_ , 2022.
  * Qasim et al. [2019] S. Qasim, H. Mahmood, and F. Shafait.  Rethinking table recognition using graph neural networks.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 142–147, Los Alamitos, CA, USA, sep 2019. IEEE Computer Society.  doi: 10.1109/ICDAR.2019.00031.  URL https://doi.ieeecomputersociety.org/10.1109/ICDAR.2019.00031.
  * Riba et al. [2019] Pau Riba, Anjan Dutta, Lutz Goldmann, Alicia Fornés, Oriol Ramos, and Josep Lladós.  Table detection in invoice documents by graph neural networks.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 122–127, 2019.  doi: 10.1109/ICDAR.2019.00028.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Wei et al. [2020] Mengxi Wei, Yifan He, and Qiong Zhang.  Robust layout-aware ie for visually rich documents with pre-trained language models, 2020.
  * Xu et al. [2021] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2579–2591, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.201.  URL https://aclanthology.org/2021.acl-long.201.
  * Xu et al. [2020] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  Layoutlm: Pre-training of text and layout for document image understanding.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1192–1200, 2020.
  * Yang et al. [2019] Huichen Yang, Carlos A. Aguirre, Maria F. De La Torre, Derek Christensen, Luis Bobadilla, Emily Davich, Jordan Roth, Lei Luo, Yihong Theis, Alice Lam, T. Yong-Jin Han, David Buttler, and William H. Hsu.  Pipelines for procedural information extraction from scientific literature: Towards recipes using machine learning and data science.  In _2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)_ , volume 2, pages 41–46, 2019.  doi: 10.1109/ICDARW.2019.10037.
  * Zhang et al. [2021] Peng Zhang, Can Li, Liang Qiao, Zhanzhan Cheng, Shiliang Pu, Yi Niu, and Fei Wu.  Vsr: A unified framework for document layout analysis combining vision, semantics and relations.  In Josep Lladós, Daniel Lopresti, and Seiichi Uchida, editors, _Document Analysis and Recognition – ICDAR 2021_ , pages 115–130, Cham, 2021\. Springer International Publishing.  ISBN 978-3-030-86549-8.
  * Zhong et al. [2019] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes.  Publaynet: largest dataset ever for document layout analysis.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1015–1022. IEEE, 2019.