11institutetext: University of Colorado Boulder
11email<EMAIL_ADDRESS>22institutetext: Electronics and
Telecommunications Research Institute
22email<EMAIL_ADDRESS>
# CPrune: Compiler-Informed Model Pruning for Efficient Target-Aware DNN
Execution

Taeho Kim 11 0000-0001-7787-3035 Yongin Kwon 22 0000-0003-2973-246X Jemin Lee
22 0000-0002-9332-3508 Taeho Kim 22 0000-0002-5061-206X Sangtae Ha 11
0000-0001-5983-5430

###### Abstract

Mobile devices run deep learning models for various purposes, such as image
classification and speech recognition. Due to the resource constraints of
mobile devices, researchers have focused on either making a lightweight deep
neural network (DNN) model using model pruning or generating an efficient code
using compiler optimization. Surprisingly, we found that the straightforward
integration between model compression and compiler auto-tuning often does not
produce the most efficient model for a target device. We propose CPrune, a
compiler-informed model pruning for efficient target-aware DNN execution to
support an application with a required target accuracy. CPrune makes a
lightweight DNN model through informed pruning based on the structural
information of subgraphs built during the compiler tuning process. Our
experimental results show that CPrune increases the DNN execution speed up to
2.73$\times$ compared to the state-of-the-art TVM auto-tune while satisfying
the accuracy requirement.

00footnotetext: Yongin Kwon is the corresponding author.

##  1 Introduction

Deep neural networks (DNNs) have become increasingly popular for various
applications like image processing and speech recognition. DNN requires heavy
computation due to large and complex neural networks, so recent approaches
proposed intelligent offloading using the cloud for resource-constrained
mobile devices. Unfortunately, they often suffer from network overhead and
disruptions, necessitating running DNNs directly on devices. Furthermore,
making DNN inferences on devices is often desirable for user experience and
privacy.

Several efforts have focused on improving DNN inference on mobile devices by
considering energy efficiency, accuracy, and execution time. These efforts are
primarily divided into two approaches: (1) making a lightweight DNN model
using model compression and (2) generating optimized code for the target
device using a DNN compiler. The model compression, for example, optimizes the
neural networks by pruning, quantization, or neural architecture search (NAS).
In particular, model pruning reduces the size of a model by removing non-
critical or redundant neurons from a DNN model [26], while quantization does
it by reducing the precision of the datatype [23]. In addition, NAS optimizes
the structure of a DNN model for maximizing performance [19]. The DNN
compiler, on the other hand, takes a DNN model and generates an optimized
execution code for each target device.

Figure 1: An experiment to find the fastest model whose accuracy is higher
than 92.80% accuracy for the target device among various pruned models of
VGG-16 for CIFAR-10. We use the number of processed figures per second (FPS),
given that indirect metrics like FLOPs cannot replace actual execution speed.
The best-pruned model does not guarantee the best after compiler optimization.
For example, the best model with pruning achieves only 2174 FPS while the
suboptimal one obtains a higher 2857 FPS after compiler optimization.

Efficient execution through a lightweight model is critical for many
applications, such as AR/VR applications, autonomous driving [24], drone or
robot control [8], medical health monitoring [16], malware detection [40], and
user authentication such as Face ID [50, 27]. While one might think that the
straightforward integration between model compression and compiler
optimization could generate the most efficient code for a specific target
device, this is not the case. Figure 1 shows experimental results of direct
integration between model compression and compiler optimization. As shown, the
fastest model meeting the required accuracy found by pruning is often not the
best model after compiler optimization, demanding joint optimization between
them.

Based on this observation, we propose CPrune, a new pruning technique for an
efficient DNN execution for target devices that jointly considers model
compression and compiler optimization. Instead of optimizing DNN compression
and compiler optimization independently, leading to suboptimal optimization,
CPrune exploits the information extracted during the compiler optimization
process. In particular, CPrune uses the subgraph structures of a neural
network and their execution times on the target device during compiler
optimization to create an efficient target-aware pruned model fulfilling
accuracy requirements. To the best of our knowledge, this is the first work
using the information collected during the compiler optimization to create the
most efficient target-aware compressed model meeting accuracy requirements.

Our contributions are:

  * •

We report from our experiments that the fastest model that meets the required
accuracy found by pruning is often not the best model after compiler
optimization, necessitating joint optimization.

  * •

We propose CPrune, which incorporates the information extracted during the
compiler optimization process into creating a target-oriented compressed model
fulfilling accuracy requirements. This information also significantly reduces
the search space for parameter tuning such that CPrune can make a compressed
model substantially faster than NetAdapt [44], the state-of-the-art hardware-
aware model compression framework.

  * •

Our experimental results show that CPrune achieves target-oriented performance
improvement and increases the speed up to 2.73$\times$ compared to the de
facto DNN compiler framework TVM auto-tune while satisfying the accuracy
requirement.

While CPrune’s approach is generally applicable, we can utilize CPrune for
computer vision tasks requiring fast processing but a slight reduction in
accuracy is acceptable, such as object detection and image classification for
autonomous driving and biosignal image processing for seizure detection on a
mobile device. We implement CPrune on top of an open deep learning compiler
stack Apache TVM [1] and Microsoft NNI [2]. Our source code can be found at
https://github.c
om/taehokim20/CPrune.

##  2 Related Works

###  2.1 Model Compression

Model compression reduces the model size and improves speed while meeting the
performance requirements: pruning compresses a model while NAS creates a much
lighter model. There are also approaches using the performance measurements on
real hardware as feedback for model compression.

Figure 2: Structured pruning removes channels or filters to reduce the model’s
overall size. In contrast, non-structured pruning selectively prunes less
important weights to minimize the sizes without impacting accuracy loss.

Pruning. Pruning removes non-critical, redundant neurons from a DNN model
[26]. Figure 2 shows an example of a convolution operation in DNN. $c_{i}$ is
the number of input channels, each of which comprises height ($h_{i}$) and
width ($w_{i}$) of the input. After convolving $c_{i}$ input channels with
filters of layer $l_{i}$, the input becomes abstracted to a feature map with
output channels $c_{i+1}$, where $c_{i+1}$ is determined by the number of
filters in layer $l_{i}$. Structured pruning [14, 15, 21, 39, 48] removes
filters or channels according to various algorithms to judge the redundancy of
weights. For example, removing a few input channels $c_{i}$ reduces the
overall sizes of the input and filter shape, and pruning a filter reduces the
sizes because it eliminates a channel in the next layer (both cases are
highlighted as shaded regions in Figure 2). On the other hand, non-structured
pruning [11, 47] selects individual weights to prune. As a result, they
achieve very high compression without much accuracy loss even though it is
hard to gain as much as speedups of structured ones [9, 10, 28].

NAS. NAS optimizes the structure of a DNN model for maximizing performance
[7]. It uses various search strategies to explore the search space of neural
architectures, such as random search, Bayesian optimization [35, 41],
evolutionary method [30, 6], reinforcement learning (RL) [51], and gradient-
based methods [7, 25].

Hardware-aware model compression. Recently, several approaches have utilized
performance measurements on real hardware to make a compressed model [22, 37,
42, 44, 45]. Especially, NetAdapt [44] points out that indirect metrics (e.g.,
the number of MACs or weights) may not necessarily reduce the direct metrics
(e.g., execution time and energy consumption) [43, 46], so they incorporate
performance metrics of target hardware into its pruning adaptation algorithm.

Note that our approach CPrune is different from NetAdapt in that it jointly
considers model compression and compiler optimization.

###  2.2 DNN Compilers with Auto-Tuning

Various DNN compilers generate code for DNN models on different hardware
architectures [4, 20, 33]. For example, TVM [4] translates the input DNN
models into the intermediate representation (IR) during the compilation
process [32]. After performing hardware-independent optimizations with the IR,
it translates high-level IR into low-level IR, which provides interfaces to
tune the computation and memory access. In addition, auto-tuning approaches
such as AutoTVM [5] and AutoScheduler [49] optimize various low-level
hardware-dependent parameters since the optimal parameters vary depending on
the DNN operations and the target hardware.

##  3 CPrune: Compiler-Informed Model Pruning

Our motivation starts from a question: Can an optimally pruned model guarantee
the best performance after compiler optimization for a specific device? To
check this question, we prune VGG-16 [36] randomly and create 20 different
models. We use the CIFAR-10 [17] dataset, and the original model’s accuracy is
93.29%. We check if the model showing the highest number of processed figures
per second (FPS) with an accuracy higher than 92.80% would show the highest
FPS after compiler optimization. We use FPS, given that indirect metrics like
FLOPs cannot replace actual execution speed. For this experiment, we run and
measure the FPS on a PC with an NVIDIA RTX-3080 GPU.

As shown in Figure 1, when we measure the FPS of each pruned model, the
fastest model that meets the accuracy requirement is model A (92.85%, 205
FPS). However, after compiler optimization on the pruned models, the fastest
model is model B (92.86%, 2857 FPS). It is different from what we expect: the
most efficient model after pruning should be the most efficient one after
compiler optimization. We also find no strong correlation between the
performance of pruned models before and after compiler optimization,
emphasizing the need for joint optimization between model compression and
compiler optimization.

Inspired by this finding, we design a new pruning scheme, CPrune, that
generates an efficient DNN execution model by pruning a model based on the
extracted information during compiler optimization for the target device.
Below we present the details of our approach.

Figure 3: CPrune overview. CPrune generates the fastest DNN execution model on
the target device by using the structural information extracted during
compiler optimization and the on-device performance of a DNN model.

###  3.1 CPrune Overview

The goal of CPrune is to find and prune neurons that primarily impact the
execution of the DNN model on the target device while meeting the accuracy
requirements. This informed pruning effectively prevents the issue of the best
pruning model not being the best on the target device after compilation.

Figure 3 depicts how CPrune leverages the information collected during
compiler optimization to prune neurons from the DNN model effectively. The
shaded boxes indicate what CPrune adds to the existing compiler optimization
framework. A DNN model consists of many convolutional layers, each
representing a subgraph ( 1). A subgraph is assigned to a task, and two
different subgraphs can point to the same task. Then, a DNN compiler creates
numerous intermediate representations for each task and selects the fastest
program on the target device ( 2). After compiling the aggregate of the
fastest programs comprising a DNN model, CPrune checks if the model meets the
minimum execution and accuracy requirements. After that, CPrune sorts tasks
based on their execution times in descending order to find the most efficient
model with further pruning. Since pruning a task with a longer execution time
could significantly reduce a model’s execution time, CPrune selects the most
time-consuming task as a candidate for further pruning ( 3). CPrune now needs
to know which subgraph(s) is associated with this task as pruning candidates.
CPrune also store the fastest program for each task. For this purpose, CPrune
builds a table keeping the relationship among tasks, subgraphs, and programs (
4). Finally, CPrune prunes subgraphs of the selected task while ensuring their
code structures follow the structure of the fastest program of that task ( 5).
Since the computation structure impacts the execution time, preserving the
same computation structure after pruning is critical for efficient pruning.
This process continues to make the most efficient pruned DNN model satisfying
the accuracy requirement.

In the following subsections, we describe the the details of an algorithm
(Section 3.2), task ordering (Section 3.3), task and subgraph relationship
table (Section 3.4) and pruning decision (Section 3.5).

###  3.2 CPrune Algorithm

CPrune prunes a DNN model gradually and iteratively. In each iteration, CPrune
analyzes subgraphs of the model and creates relationships between subgraphs
and each task. Then, it takes a pre-trained model $M$ and the minimum accuracy
requirement $a_{g}$ and returns the efficient target-aware DNN executable
program, as shown in Algorithm 1. The computational complexity of Algorithm 1
and a table that summarizes variables used are in the Supplementary Materials.

Initialization step (Line 1). It initializes tuning-related parameters,
including the pruning rate of each subgraph $p_{r}$, the target execution time
of the following iteration $l_{t}$, and the short-term accuracy of the
previous best model $a_{p}$ in Algorithm 1. It also initializes a
task/subgraph table $C$, storing the relationship among tasks, subgraphs, and
fastest programs and the list of tasks prioritized by tuning $R$.

Main step: pruning based on the compiler optimization (Line 2-16). In each
pruning iteration (the while loop in Algorithm 1) of this step, CPrune
generates a pruned candidate model ( 1 in Figure 3) and moves to the
intermediate representation (IR) and tuning stage ( 2). When a task in order
of $R$ is picked up ( 3), CPrune extracts associated subgraphs and the fastest
program of the task from $C$ ( 4). It decides the number of filters to prune
by analyzing the arrangement of filters of the program and prunes filters of
the subgraphs to create a pruned candidate model $M^{\prime}$ ( 5, details in
Section 3.5). CPrune uses the relationship between subgraphs of the model
$M^{\prime}$ and each task and creates a task/subgraph table $C^{\prime}$. It
also puts the fastest program of each task in $C^{\prime}$ (details in Section
3.4). Furthermore, the IR and tuning process maintains a candidate list of
prioritized tasks $R^{\prime}$ for the next iteration (details in Section
3.3). If the measured execution time of $M^{\prime}$ ($l_{m}$) is less than
$l_{t}$, it trains $M^{\prime}$ shortly and measures the short-term accuracy
$a_{s}$. When $a_{s}$ meets the requirement of the current iteration, it
updates parameters and goes to the next iteration. $\alpha$ is the ratio to
represent the minimum allowable accuracy after pruning, and $\beta$ is the
ratio to define the target execution time of the next pruning iteration.

Algorithm 1 CPrune Algorithm

Input: Pre-trained model $M$ and $a_{g}$
Output: An efficient DNN executable file of $M$

1:Tune $M$ and initialize $p_{r}$, $l_{t}$, $a_{p}$, $C$ and $R$

2:while $a_{p}>a_{g}$ and $R\neq\\{\\}$ do

3: for $r$ in $R$ do

4: For $r$, obtain associated subgraphs ($S$) and the fastest program ($P$)
from $C$

5: Update pruning rate ($p_{r}$) by analyzing the structure of $P$

6: Create a pruned candidate model ($M^{\prime}$) by pruning $S$ with pruning
rate $p_{r}$

7: Extract tasks from $M^{\prime}$ and create a candidate task/subgraph table
($C^{\prime}$)

8: Tune these tasks and create a candidate list of prioritized tasks
($R^{\prime}$)

9: Compile the aggregate of tuned tasks and measure $l_{m}$

10: Go line 3 if $l_{m}\geq l_{t}$

11: Short-term train $M^{\prime}$ and measure $a_{s}$

12: Remove $r$ from $R$ and go line 3 if $a_{s}<\alpha\cdot a_{p}$

13: $M\leftarrow M^{\prime}$, $R\leftarrow R^{\prime}$, $C\leftarrow
C^{\prime}$, update $p_{r}$, $l_{t}=\beta\cdot l_{m}$, $a_{p}=a_{s}$

14: break

15: end for

16:end while

17:Final long-term training, tuning and compilation for $M$

If $l_{m}$ is greater than or equal to $l_{t}$, the system selects the next
task from $R$ to create the next pruned candidate model. Once $a_{s}$ is less
than the target accuracy of the current iteration $(\alpha\cdot a_{p}$),
CPrune removes the current task from $R$ and does not consider this task as a
candidate for pruning in the rest iterations.

Final step: final training and tuning (Line 17). If there are no more tasks in
$R$ that can be pruned while meeting the accuracy requirement, CPrune
progresses to the final step. In this step, CPrune trains and tunes the final
model to achieve optimal accuracy and execution time.

###  3.3 Task Ordering

CPrune needs to select the most promising task of the current model $M$ for
pruning. Therefore, CPrune sorts tasks according to the pruning impact (task’s
execution time $\times$ number of subgraphs associated with the task). The
higher the pruning impact, the more likely the task satisfies the current
iteration’s target execution time $l_{t}$.

The rightmost shaded box in Figure 3 shows the result of task ordering as an
example. The pruning impact of $T_{1}$ is $0.954\times 2=1.908$, $T_{2}$ is
$0.473\times 3=1.419$, $T_{3}$ is $1.632\times 1=1.632$, and so on. After
CPrune performs task ordering including the results of other tasks, the
ordered list $R^{\prime}$ becomes {$T_{1}$, $T_{3}$, $T_{2}$, …, $T_{n}$, …}.
In this example, the subgraphs associated to $T_{1}$ will be pruned first. If
this choice does not satisfy the execution time and accuracy requirements,
$T_{3}$ and then $T_{2}$ will be selected for the next pruning candidate.

###  3.4 Task and Subgraph Relationship

Figure 4: The graph analysis of CPrune for the ResNet-18 model

CPrune creates a table storing the relationship among tasks, subgraphs, and
fast programs to determine which subgraphs are pruned and which program is
analyzed when selecting a task in the next pruning iteration. CPrune analyzes
a pruned candidate model by checking the layer connectivity, weight shape,
Rectified Linear Unit (ReLU), and Batch Normalization (BN). Figure 4 shows
CPrune’s graph analysis for part of the ResNet-18 model [3]. As ResNet-18
consists of multiple convolutions of the same shape, the compiler partitions
the large computational graph of a DNN into multiple subgraphs, each of which
can be associated with other subgraphs [32]. For example, subgraphs $S_{11}$
and $S_{14}$ are connected to the same task $T_{7}$ due to the same properties
of BN and input shapes. On the other hand, subgraphs $S_{10}$, $S_{12}$, and
$S_{13}$ are connected to different tasks, respectively. In addition, each
task is connected to its fastest program during the tuning.

Figure 5: Sampled program comparison extracted from one subgraph among several
subgraphs of ResNet-18. $c_{i}$ is the number of input channels.

###  3.5 Pruning Decision

CPrune prunes filters while maintaining the structure of the fastest program
created during the IR tuning process of compiler optimization. Figures 5 (b)
and (c) compare the structures of the fastest and slowest programs generated
from the same subgraph in Figure 5 (a). As we can see, the fastest program
Figure 5 (b) allows a subgraph to perform parallel processing of computations
most efficiently. In contrast, the slowest program Figure 5 (c) shows the most
inefficient parallel processing of computations. If CPrune prunes four filters
suitable for maintaining the structure of the program Figure 5 (c), a DNN
compiler may generate an inefficient program for the corresponding subgraph of
a DNN model. Instead, pruning the subgraph to follow the program structure of
the fastest program would lead a DNN compiler to generate an efficient
program. In addition, since the execution time of the convolution layer
increases in a step pattern rather than linear with the number of filters
[38], pruning an insufficient number of filters would only increase the tuning
time without improving FPS.

In particular, CPrune splits a given model into multiple subgraphs and
determines the number of filters to prune for each subgraph. We elaborate on
calculating the number of filters to be pruned by taking one subgraph as an
example, as shown in Figure 5 (a). We can see that the total number of filters
is 512, and the kernel shape is 7 $\times$ 7 like Figure 5 (d) by checking the
2D convolution layer (Conv2d) and the ReLU activation function layer (T_relu)
in Figure 5 (a). Iterators related to the number of filters are ff and ax3.
With this information, CPrune checks how these iterators are further split or
merged. For example, Figure 5 (b) shows that 512 of ff in Figure 5 (a) is
converted to 4$\times$8$\times$16 by the argument ff.3. 512 of ax3 is also
converted to 4$\times$8$\times$16\. Therefore, both iterators arrange filters
like Figure 5 (e).

Finally, CPrune calculates the minimum number of filters to prune based on the
arrangement shapes of filters obtained from the related iterators. If CPrune
reduces the number in width, height, or depth in the arrangement of filters of
Figure 5 (e), the subgraph can maintain the structure of Figure 5 (b) after
pruning the filters. However, if CPrune prunes many filters at once, it cannot
achieve optimal pruning due to the high difference from the performance goal
in the current pruning iteration. Therefore, we prune filters by the step size
to improve performance by pruning in the convolution layer [38]. Let
$L_{1}=\\{a_{1},a_{2},...,a_{m}\\}$ be the set of product combinations of the
first iterator, and $L_{2}=\\{b_{1},b_{2},...,b_{n}\\}$ be the set of product
combinations of the second iterator. CPrune determines the least common
multiple (LCM) for the two pruning numbers as the minimum number of pruning
filters using $L_{1}$ and $L_{2}$.

$\displaystyle LCM(\min_{l\in L_{1}}\frac{\prod_{k=1}^{m}a_{k}}{l},\min_{l\in
L_{2}}\frac{\prod_{k=1}^{n}b_{k}}{l})=LCM(\frac{\prod_{k=1}^{m}a_{k}}{\max_{l\in
L_{1}}l},\frac{\prod_{k=1}^{n}b_{k}}{\max_{l\in L_{2}}l})$

For example, we calculate the minimum number of filters that can be pruned
while maintaining the program structure in Figure 5 (b) and Figure 5 (c),
respectively. In Figure 5 (b), the product combination of iterators ff and ax3
is the same as 4$\times$8$\times$16\. CPrune can prune 8$\times$16,
4$\times$16, or 4$\times$8 filters like Figure 5 (f) while maintaining the
product combination. Thus, the minimum number of pruning filters of Figure 5
(b) is $LCM(\min\\{8\times 16,4\times 16,4\times 8\\},\min\\{8\times
16,4\times 16,4\times 8\\})=32$. On the other hand, in Figure 5 (c), ff and
ax3 are converted to 4$\times$128 and 512$\times$1, respectively. Therefore,
the minimum number of pruning filters of Figure 5 (c) is
$LCM(\min\\{4,128\\},\min\\{512,1\\})=4$.

After determining the number of filters to be pruned, CPrune decides which
filters to prune. It calculates the sum of each filter’s absolute weights
(i.e., $l_{1}$-norm) and prunes filters starting with the smallest sum [2,
21].111Using other metrics can improve the performance as well. CPrune also
prunes the input channels of the next layer by the determined number as
described in Section 2.1 to maintain the connection consistency. In the case
of the shortcut part in a ResNet model, we prune the same output channels
using the dependency-aware mode supported by Microsoft NNI [2].

##  4 Experiments

###  4.1 Experimental Setup

In this section, we carry out various experiments to evaluate the performance
of CPrune. We conduct experiments over different pre-trained DNN models,
including ResNet-18 [12], MobileNetV2 [34], and MnasNet1.0 [37], with ImageNet
[18] or CIFAR-10 [17] datasets on various resource-constrained mobile devices
(Samsung Galaxy S8 (Kryo 280 CPU), S9 (Kryo 385 CPU and Mali-G72 GPU), S20+
(Kryo 585 CPU), or Google Pixel 3 XL (Kryo 385 CPU)). We also use multiple
host PCs with NVIDIA GeForce RTX 1080 Ti or 2080 Ti. All pruned models are
optimized by stochastic gradient descent (SGD) [31]. The number of training
epochs considered varies according to the choice of the dataset and the
specific phase of training (i.e., either short-term or final training). For
the CIFAR-10 dataset, the short-term and final training epochs are 5 and 100,
respectively. For the ImageNet dataset, the training epochs are one-fifth of
CIFAR-10 due to the enormous data size. Due to space limitations, the details
of finding reasonable $\alpha$ and $\beta$ values and additional experiments
about the impact of tuning and selective search are in the Supplementary
Materials.

(a) Short-term accuracy (b) FPS increase rate

Figure 6: The FPS increase rate and short-term accuracy during the iterative
CPrune process compared to using only TVM auto-tune. After completing the
compiler-informed pruning process, CPrune performs final long-term training
and tuning (Line 17 of Algorithm 1). Therefore, we marked the final results
with dotted lines to distinguish them from other short-term measurements.

###  4.2 Overall Performance

This experiment shows CPrune’s FPS and short-term accuracy during the
iterative pruning process compared to the case of using only a DNN compiler
and its auto-tuning. For this experiment, we select TVM auto-tune [49] for the
DNN compiler and use the ResNet-18 model on Kyro 385 CPU using the ImageNet
dataset. Figure 6 shows the results. In each pruning iteration, CPrune
modifies the current model by pruning selected subgraphs. For any given
pruning iteration, if a pruned candidate model satisfies the condition of the
given iteration ($l_{t}$ and $\alpha\cdot a_{p}$), the candidate model is
selected, and the iteration progresses. The final accuracy is 88.34%, and the
FPS increase rate is 1.96 times faster than using TVM auto-tune alone. We
emphasize that our CPrune’s FPS increase rate is around two times that
obtained by only using the TVM auto-tune, while the final accuracy is within
tolerable limits. Note that CPrune can stop its pruning around the $6^{th}$
pruning iteration if the accuracy requirement is more than 89%. Then the FPS
increase rate is 1.3 times faster than TVM auto-tune.

In a practical scenario, user applications can provide the accuracy and
execution time requirements to CPrune.

###  4.3 Performance on Different Target Devices

This section evaluates CPrune’s performance on different target devices. For
this experiment, we integrate CPrune with the TVM compiler to compare its
performance on different target devices. We also convert the final pruned
model to a widely used TFLite executable for performance comparison. We
compare the FPS of different DNN models using the ImageNet dataset on mobile
CPU (Kryo 385 or Kryo 585) and GPU (Mali-G72), as shown in Figure 7. CPrune,
along with the target compiler framework, shows a significantly higher FPS
than the cases of running only a DNN compiler (e.g., TVM) and a target
agnostic DNN library (e.g., TFLite). Furthermore, regardless of the type of
processor and model, the FPS increase rate when executing the CPrune model on
a target processor is significantly higher than when we run it on other
processors, as shown in Figure 8.

Figure 7: Comparison of FPS when we execute CPrune model with the target
compiler framework TVM and a target-agnostic deep learning framework TFLite.
Figure 8: Comparison of FPS when we execute CPrune model on a target processor
and other processors.

###  4.4 Comparison with Other Pruning Schemes

This section compares CPrune with other pruning schemes on different mobile
platforms. We also integrate CPrune into TVM for a fair comparison. Table 4.4
shows the results. CPrune shows a higher FPS than the model-based pruning
models (e.g., PQF [29], FPGM [13], AMC [14]). CPrune also shows similar or
better performance than the hardware-aware pruning model (e.g., NetAdapt [44])
with TVM. It also shows that indirect metrics such as FLOPS and parameters do
not fully reflect actual performance. While FLOPS is a suitable indirect
measure of the extent of compression obtained during the pruning process, FPS
suitably reflects the pruning gains in terms of execution times or speeds.

Table 1: Mobile CPU (Kryo 385 and 585) and GPU (Mali-G72) performance test
(ResNet-18, MobileNetV2, MnasNet1.0 with ImageNet dataset)

Model |  Method |  FPS  (Increase rate) |  FLOPS |  Params |  Top-1 Acc |  Top-5 Acc
---|---|---|---|---|---|---
ResNet-18  (Kryo 385) |  Original (TVM) |  18.86 |  1.81B |  11.7M |  69.76% |  89.08%
PQF [29]+TVM |  18.73 (0.99$\times$) |  166M |  8.26M |  66.74% |  87.16%
FPGM [13]+TVM |  22.93 (1.22$\times$) |  1.10B |  7.30M |  68.37% |  88.43%
NetAdapt [44]+TVM |  35.17 (1.86$\times$) |  1.24B |  9.59M |  68.45% |  88.37%
CPrune |  36.92 (1.96$\times$) |  1.17B |  10.3M |  68.30% |  88.34%
ResNet-18 (Mali-G72) |  Original |  15.65 |  1.81B |  11.7M |  69.76% |  89.08%
PQF+TVM |  24.14 (1.54$\times$) |  166M |  8.26M |  66.74% |  87.16%
FPGM+TVM |  26.62 (1.70$\times$) |  1.10B |  7.30M |  68.37% |  88.43%
CPrune |  30.02 (1.92$\times$) |  1.55B |  10.4M |  69.83% |  89.24%
MobileNetV2 (Kryo 385) |  Original |  28.20 |  301M |  3.47M |  71.88% |  90.29%
AMC [14]+TVM |  67.62 (2.40$\times$) |  211M |  2.31M |  70.85% |  89.91%
CPrune |  76.92 (2.73$\times$) |  255M |  3.29M |  70.33% |  89.57%
MobileNetV2 (Mali-G72) |  Original |  68.68 |  301M |  3.47M |  71.88% |  90.29%
AMC+TVM |  90.58 (1.32$\times$) |  211M |  2.31M |  70.85% |  89.91%
CPrune |  101.56 (1.48$\times$) |  281M |  3.31M |  71.39% |  90.16%
MnasNet1.0  (Kryo 585) |  Original |  42.92 |  314M |  4.35M |  73.46% |  91.51%
CPrune |  56.85 (1.32$\times$) |  284M |  3.82M |  72.90% |  91.16%

###  4.5 Effect of Pruning on Associated Subgraphs

This experiment checks the effectiveness of pruning filters of all subgraphs
related to a task as in CPrune than pruning filters of only one subgraph in
each iteration. For a task associated with greater than one subgraph, we have
a design choice to prune filters of a single subgraph at a time (NetAdapt
[44]) or prune filters of all the subgraphs associated with the task (like
CPrune). Associated subgraphs pruning can shorten the CPrune process in
proportional to the number of related subgraphs in the model. We observe that
the associated subgraphs pruning consumes relatively less time in the Main
step of CPrune than the single subgraph pruning, as shown in Figure 9 (a). In
addition, the associated subgraphs pruning improves the FPS by more than 13
FPS compared to the single subgraph pruning strategy without significantly
reducing the accuracy, as shown in Figure 9 (b) and Table 4.5.

Table 2: Mobile CPU performance test (ResNet-18 with CIFAR-10 dataset)

Model |  Method |  FPS (Increase rate) |  FLOPS |  Params |  Top-1 Acc
---|---|---|---|---|---
ResNet-18  (Kryo 280) |  Original (TVM) |  33.82 |  555M |  11.2M |  94.37%
CPrune |  109.45 (3.24$\times$) |  161M |  2.62M |  93.74%
ResNet-18  (Kryo 585) |  Original |  40.50 |  555M |  11.2M |  94.37%
CPrune |  93.63 (2.31$\times$) |  297M |  3.54M |  94.14%
CPrune (w/o tuning) |  57.77 (1.43$\times$) |  390M |  5.08M |  94.51%
CPrune (single subgraph pruning) |  79.62 (1.97$\times$) |  294M |  4.55M |  94.27%

(a) Relative time cost (b) FPS

Figure 9: The effect of pruning all the subgraphs associated with the task
(ResNet-18, Kryo 585, CIFAR-10). (a) is the relative time cost comparison with
pruning a single subgraph at a time

##  5 Conclusion

Existing methods generate a compressed model by focusing on a model itself for
fast DNN model execution on resource-constrained target devices. However, we
have confirmed that knowing which model is best from pruning is impossible
without considering compiler optimization. Therefore, we propose CPrune, a
compiler-informed model pruning for efficient target-aware DNN execution.
CPrune ensures the actual performance with pruning on the target device by
using the relationship between the task and subgraphs during the compiler
optimization. CPrune generates a target-aware DNN execution model by pruning a
model based on on-device performance and compiler optimization. We verified
that the pruned model generated by CPrune improves a DNN model’s execution
speed significantly while meeting the accuracy requirement.

Acknowledgments: This work was supported by the Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No. 2018-0-00769, Neuromorphic Computing Software
Platform for Artificial Intelligence Systems).

## References

  * [1] Apache tvm. https://github.com/apache/tvm, accessed: 2022-07-03
  * [2] Microsoft nni. https://github.com/microsoft/nni, accessed: 2022-07-03
  * [3] Torchvision models. https://pytorch.org/vision/stable/models.html, accessed: 2022-07-03
  * [4] Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al.: $\\{$TVM$\\}$: An automated end-to-end optimizing compiler for deep learning. In: 13th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 18). pp. 578–594 (2018)
  * [5] Chen, T., Zheng, L., Yan, E., Jiang, Z., Moreau, T., Ceze, L., Guestrin, C., Krishnamurthy, A.: Learning to optimize tensor programs. arXiv preprint arXiv:1805.08166 (2018)
  * [6] Chen, Y., Meng, G., Zhang, Q., Xiang, S., Huang, C., Mu, L., Wang, X.: Renas: Reinforced evolutionary neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4787–4796 (2019)
  * [7] Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. The Journal of Machine Learning Research 20(1), 1997–2017 (2019)
  * [8] Fang, B., Zeng, X., Zhang, M.: Nestdnn: Resource-aware multi-tenant on-device deep learning for continuous mobile vision. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. pp. 115–127 (2018)
  * [9] Gale, T., Zaharia, M., Young, C., Elsen, E.: Sparse gpu kernels for deep learning. IEEE Press (2020)
  * [10] Gong, Z., Ji, H., Fletcher, C.W., Hughes, C.J., Baghsorkhi, S., Torrellas, J.: Save: Sparsity-aware vector engine for accelerating dnn training and inference on cpus. In: 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). pp. 796–810 (2020)
  * [11] Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient dnns. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. pp. 1387–1395 (2016)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [13] He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4340–4349 (2019)
  * [14] He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., Han, S.: Amc: Automl for model compression and acceleration on mobile devices. In: Proceedings of the European conference on computer vision (ECCV). pp. 784–800 (2018)
  * [15] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE international conference on computer vision. pp. 1389–1397 (2017)
  * [16] Kim, T., Nguyen, P., Pham, N., Bui, N., Truong, H., Ha, S., Vu, T.: Epileptic seizure detection and experimental treatment: a review. Frontiers in Neurology 11, 701 (2020)
  * [17] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [18] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25, 1097–1105 (2012)
  * [19] Kyriakides, G., Margaritis, K.: An introduction to neural architecture search for convolutional networks. arXiv preprint arXiv:2005.11074 (2020)
  * [20] Lattner, C., Amini, M., Bondhugula, U., Cohen, A., Davis, A., Pienaar, J., Riddle, R., Shpeisman, T., Vasilache, N., Zinenko, O.: Mlir: Scaling compiler infrastructure for domain specific computation. In: 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). pp. 2–14. IEEE (2021)
  * [21] Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
  * [22] Li, Z., Yuan, G., Niu, W., Zhao, P., Li, Y., Cai, Y., Shen, X., Zhan, Z., Kong, Z., Jin, Q., et al.: Npas: A compiler-aware framework of unified network pruning and architecture search for beyond real-time mobile acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14255–14266 (2021)
  * [23] Liang, T., Glossner, J., Wang, L., Shi, S., Zhang, X.: Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461, 370–403 (2021)
  * [24] Liu, H., He, Y., Yu, F.R., James, J.: Flexi-compression: A flexible model compression method for autonomous driving. In: Proceedings of the 11th ACM Symposium on Design and Analysis of Intelligent Vehicular Networks and Applications. pp. 19–26 (2021)
  * [25] Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
  * [26] Liu, J., Tripathi, S., Kurup, U., Shah, M.: Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey. arXiv preprint arXiv:2005.04275 (2020)
  * [27] Lu, L., Yu, J., Chen, Y., Liu, H., Zhu, Y., Kong, L., Li, M.: Lip reading-based user authentication through acoustic sensing on smartphones. IEEE/ACM Transactions on Networking 27(1), 447–460 (2019)
  * [28] Ma, X., Guo, F.M., Niu, W., Lin, X., Tang, J., Ma, K., Ren, B., Wang, Y.: Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 5117–5124 (2020)
  * [29] Martinez, J., Shewakramani, J., Liu, T.W., Bârsan, I.A., Zeng, W., Urtasun, R.: Permute, quantize, and fine-tune: Efficient compression of neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15699–15708 (2021)
  * [30] Periaux, J., Gonzalez, F., Lee, D.S.C.: Evolutionary methods. In: Evolutionary Optimization and Game Strategies for Advanced Multi-Disciplinary Design, pp. 9–20. Springer (2015)
  * [31] Robbins, H., Monro, S.: A stochastic approximation method. The annals of mathematical statistics pp. 400–407 (1951)
  * [32] Roesch, J., Lyubomirsky, S., Weber, L., Pollock, J., Kirisame, M., Chen, T., Tatlock, Z.: Relay: A new ir for machine learning frameworks. In: Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. pp. 58–68 (2018)
  * [33] Rotem, N., Fix, J., Abdulrasool, S., Catron, G., Deng, S., Dzhabarov, R., Gibson, N., Hegeman, J., Lele, M., Levenstein, R., et al.: Glow: Graph lowering compiler techniques for neural networks. arXiv preprint arXiv:1805.00907 (2018)
  * [34] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)
  * [35] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., De Freitas, N.: Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE 104(1), 148–175 (2015)
  * [36] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [37] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2820–2828 (2019)
  * [38] Tang, X., Han, S., Zhang, L.L., Cao, T., Liu, Y.: To bridge neural network design and real-world performance: A behaviour study for neural networks. Proceedings of Machine Learning and Systems 3, 21–37 (2021)
  * [39] Wang, Z., Li, C., Wang, X.: Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14913–14922 (2021)
  * [40] Wei, L., Luo, W., Weng, J., Zhong, Y., Zhang, X., Yan, Z.: Machine learning-based malicious application detection of android. IEEE Access 5, 25591–25601 (2017)
  * [41] White, C., Neiswanger, W., Savani, Y.: Bananas: Bayesian optimization with neural architectures for neural architecture search. arXiv preprint arXiv:1910.11858 1(2) (2019)
  * [42] Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10734–10742 (2019)
  * [43] Yang, T.J., Chen, Y.H., Sze, V.: Designing energy-efficient convolutional neural networks using energy-aware pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5687–5695 (2017)
  * [44] Yang, T.J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., Adam, H.: Netadapt: Platform-aware neural network adaptation for mobile applications. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 285–300 (2018)
  * [45] Yang, T.J., Liao, Y.L., Sze, V.: Netadaptv2: Efficient neural architecture search with fast super-network training and architecture optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2402–2411 (2021)
  * [46] Yu, J., Lukefahr, A., Palframan, D., Dasika, G., Das, R., Mahlke, S.: Scalpel: Customizing dnn pruning to the underlying hardware parallelism. ACM SIGARCH Computer Architecture News 45(2), 548–560 (2017)
  * [47] Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., Wang, Y.: A systematic dnn weight pruning framework using alternating direction method of multipliers. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 184–199 (2018)
  * [48] Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2780–2789 (2019)
  * [49] Zheng, L., Jia, C., Sun, M., Wu, Z., Yu, C.H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., et al.: Ansor: Generating high-performance tensor programs for deep learning. In: 14th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 20). pp. 863–879 (2020)
  * [50] Zhou, B., Lohokare, J., Gao, R., Ye, F.: Echoprint: Two-factor authentication using acoustics and vision on smartphones. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. pp. 321–336 (2018)
  * [51] Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)


# Exploring prompts to elicit memorization in masked language model-based
named entity recognition

Yuxi Xia1,3, Anastasiia Sedova1,3, Pedro Henrique Luz de Araujo1,3, Vasiliki
Kougia1,3,
Lisa Nußbaumer2, Benjamin Roth1,2
1Faculty of Computer Science, University of Vienna, Vienna, Austria.
2Faculty of Philological and Cultural Studies, University of Vienna, Vienna,
Austria.
3UniVie Doctoral School Computer Science, Vienna, Austria.
{yuxi.xia<EMAIL_ADDRESS>

###### Abstract

Training data memorization in language models impacts model capability
(generalization) and safety (privacy risk). This paper focuses on analyzing
prompts’ impact on detecting the memorization of 6 masked language model-based
named entity recognition models. Specifically, we employ a diverse set of 400
automatically generated prompts, and a pairwise dataset where each pair
consists of one person’s name from the training set and another name out of
the set. A prompt completed with a person’s name serves as input for getting
the model’s confidence in predicting this name. Finally, the prompt
performance of detecting model memorization is quantified by the percentage of
name pairs for which the model has higher confidence for the name from the
training set. We show that the performance of different prompts varies by as
much as 16 percentage points on the same model, and prompt engineering further
increases the gap. Moreover, our experiments demonstrate that prompt
performance is model-dependent but does generalize across different name sets.
A comprehensive analysis indicates how prompt performance is influenced by
prompt properties, contained tokens, and the model’s self-attention weights on
the prompt.

##  1 Introduction

Recent studies have highlighted the memorization of training data in language
models (Neel & Chang, 2023), especially for auto-regressive models
(Mireshghallah et al., 2022; Carlini et al., 2023). Such studies are important
in exploring model capability (generalization (Feldman, 2020)) or safety
(privacy risk (Neel & Chang, 2023)). However, few works (Ali et al., 2022)
explore memorization in Masked Language Model (MLM)-based Named Entity
Recognition (NER) models, despite their state-of-the-art performances in
different domains, including health records (Richie et al., 2023), social
media (Yu et al., 2023), and legal documents (Darji et al., 2023).

Unlike Ali et al. (2022), who use only 5 hand-written prompts to detect model
memorization of a self-fine-tuned NER model on their private training data
(not publicly accessible), we consider the sensitivity of model confidence
towards individual prompt variations (Feng et al., 2024), and thus employ a
set of 400 diverse prompts generated by a generative language model (i.e.,
ChatGPT (OpenAI, 2023)). The prompt set covers 4 types of prompts (i.e.,
declarative, exclamatory, imperative, and interrogative), 15 prompt token
lengths, and 10 token positions for holding the target entity (person’s name).
We focus on exploring the prompts’ impact on detecting the memorization of
persons’ names in 6 publicly accessible NER models fine-tuned on the
CoNLL-2003 dataset (Tjong Kim Sang & De Meulder, 2003).

To quantify model memorization, we create a pairwise dataset sampled from
Wikidata (Vrandečić & Krötzsch, 2014) consisting of a large number
($826^{2}+825^{2}$) of person’s name (PER) pairs. This dataset is divided into
development (dev) and test sets, each containing different named entities
(PERs). Every instance in both sets is a pair of PERs, composed of an In-train
PER (from the training data of the NER model) and an Out-train PER (not
present in the training data).

Figure 1 illustrates the full pipeline for model memorization detection.
Specifically, each prompt from the prompt set is completed separately with all
PER entities from the pairwise dataset. A completed sentence, consisting of a
prompt and a $PER$, serves as input to a NER model to get the confidence score
of the $PER$ in this prompt. We define that given an In-train and Out-train
PER pair, the pair exposes sample memorization (S-MEM) if the In-train PER
receives a higher confidence score than the Out-train PER. Ultimately, the
prompt’s performance in detecting model memorization is quantified by the
percentage of name pairs that expose sample memorization in the pairwise
dataset.

Overall, our contributions are:

  * •

We are the first to analyze the prompt’s influence on the memorization
detection of MLM-based NER models, using 400 diverse prompts and 6 publicly
accessible NER models, and we study their impact across different model
properties.

  * •

We show that memorization detection is sensitive to the prompt choice.
Performance differences of prompts in the set on the same model can be as
large as 16 percentage points. Prompt performance is model-dependent but
generalizes across mutually exclusive name sets (from dev to test data).

  * •

We apply ensembling and prompt engineering techniques on the prompt set and
observe that the engineered (modified) prompts can improve performance by up
to 2 percentage points compared to the best-performed prompt in the prompt
set.

  * •

We comprehensively analyze how prompt performance is influenced by various
factors, including the properties of the prompt, the tokens it contains, and
the self-attention weights of the model focused on the prompt.

Figure 1: Memorization detection of a NER model using diverse prompts and the
pairwise dataset. $P^{In}_{*}$ and $P^{Out}_{*}$ refer to a person’s name in
and out of the training data respectively.

##  2 Related Work

Memorization in auto-regressive language models was first studied by Carlini
et al. (2019) who showed that LSTMs memorize a significant fraction of their
training data. Later, Carlini et al. (2021) and Carlini et al. (2023) use
extractability to measure the memorization of GPT-2 and GPT-Neo models Wei et
al. 2023, showing that prefixed prompts can extract sensitive training data.
However, extractability only applies to auto-regressive language models that
use only the input side of fine-tuning data, but not to fine-tuned MLMs that
use also the output side (input labels). Several studies (Feldman, 2020; Brown
et al., 2021; Razeghi et al., 2022) indicate that a certain amount of
memorization is not necessarily undesirable but helpful for model
generalization. Tirumala et al. (2022) and Magar & Schwartz (2022) study
memorization in MLMs towards the model generalization using a metric as same
as accuracy and accuracy difference between training and test sets
respectively. Ali et al. (2022) focus on the memorization in fine-tuned MLMs
(NER models) and average the confidence ranks of In-train entities in an
entity set to measure model memorization. However, Ali et al. (2022) self-
fine-tune the NER model and study the impact of the training data duplication
and complexity on the memorization dynamics. Differently, we explore 6
publicly accessible NER models fine-tuned by different developers and focus on
the prompts’ impact on the memorization detection of the NER models.

Hand-written vs Generated Prompts. Language model performance has been shown
to be sensitive to prompt variations in previous works (Feng et al. 2024;
Sclar et al. 2023). Thus, randomly choosing one of the 5 hand-written prompts
for each studied entity to detect memorization as in Ali et al. (2022) may not
accurately represent the model memorization. Instead, we study 400 generated
diverse prompts, showing the big variances of memorization values detected by
different prompts individually were not considered in previous settings.

##  3 Methodology

In this section, we first describe the processes for creating the pairwise
dataset and prompt set. Then, we introduce how to use them to detect
memorization in NER models.

###  3.1 Pairwise Dataset Creation

We create a pairwise dataset $D_{pw}$ to quantify the memorization of NER
models fine-tuned on the CoNLL-2003 dataset (Tjong Kim Sang & De Meulder,
2003). Specifically, we first generate datasets $D_{t}$ and $D_{W}$ which
contain all $PER$s from the CoNLL-2003 dataset and Wikidata (Vrandečić &
Krötzsch, 2014) respectively. $D_{pw}$ is then sampled from the $PER$s in
$D_{W}$ considering the $PER$s in $D_{t}$. The details of the datasets are
shown in Table 1:

NER-train $PER$ Dataset ($D_{t}$). CoNLL-2003 is a commonly used NER dataset
with a large number of examples of named entities. The training dataset of the
CoNLL-2003 corpus contains 6,600 $PER$s, i.e., entities with ground-truth
label B-PER and/or I-PER which denote the beginning and the rest of a person’s
name respectively. After the post-processing of removing the duplicates and
single-token $PER$, $D_{t}$ contains 2,645 unique multi-token $PER$s.

Wikidata $PER$ Dataset ($D_{W}$). Wikidata as a source for $D_{W}$ enables the
compilation of a list of real-world person names. Specifically, a SPARQL query
is formulated to retrieve $PER$ entities. Similar to $D_{t}$, duplicates and
single-token $PER$ are filtered out. Finally, 7,617,797 multi-token $PER$ form
the dataset $D_{W}$.

Pairwise Dataset ($D_{pw}$). We only sample the $D_{pw}$ from $D_{W}$ to
ensure same data source for both In-&Out-train $PER$s. The intersection of
$D_{W}$ and $D_{t}$ ($D_{W}\cap D_{t}$) contains 1,651 $PER$s, which
onstitutes to the In-train $PER$ set $\\{P^{In}_{i}\\}^{N}_{i=1}$, N=1,651. In
contrast, an equal size of Out-train $PER$ set $\\{P^{Out}_{j}\\}^{N}_{j=1}$
are sampled from $D_{W}$ excluding $D_{t}$ ($D_{W}\setminus D_{t}$). As there
is no training process in our setting, we split $\\{P^{In}_{i}\\}^{N}_{i=1}$
and $\\{P^{Out}_{j}\\}^{N}_{j=1}$ sets equally into development (dev) and test
respectively. In the end, the dev data is constructed with
$\\{P^{In}_{i}\\}^{n}_{i=1}$ and $\\{P^{Out}_{j}\\}^{n}_{j=1}$, $n=826(\approx
N/2)$, the rest of the $PER$s goes to the test data. The test data is used to
verify the generalization of prompt performance across different name sets
(from dev to test). Each instance in dev and test sets is a name pair
consisting of a In-train $PER$ and a Out-train $PER$, and every $PER$ in
$\\{P^{In}_{i}\\}^{n}_{i=1}$ is paired with all the $PER$ in
$\\{P^{Out}_{j}\\}^{n}_{j=1}$, resulting in $n^{2}$ instances in the dev data
of $D_{pw}$.

Dataset | Source data | #PER | #Instance | Property
---|---|---|---|---
$D_{t}$ | CoNLL-2003 | 2,645 | 2,645 | In-train
$D_{W}$ | Wikidata | 7,617,797 | 7,617,797 |  In- & Out-train
$D_{pw}$ |  $D_{W}\cap D_{t}$ \+ $D_{W}\setminus D_{t}$ | 1,651 + 1,651 |  $826^{2}$(dev) + $825^{2}$(test) |  In- & Out-train
Table 1: Datasets details.

###  3.2 Prompt Set Generation

To collect a large and diverse set of prompts, we use a generative large
language model, GPT-3.5-turbo-1106 (OpenAI, 2023) (ChatGPT), to automatically
generate $m$ ($m=400$) diverse prompts $\\{PT_{k}\\}_{k=1}^{m}$. Specifically,
to increase the diversity of the prompts, we consider four categories of
sentences: (1) declarative; (2) exclamatory; (3) imperative; and (4)
interrogative. Next, we prompted ChatGPT to generate 100 sentences for each
category. To collect exclamatory prompts, we use the following prompt:
“Generate 100 different exclamatory sentences that must contain persons’
names, and replace the person’s name with ’MASK”’. The prompts of other
categories are generated by replacing the “exclamatory” string. The “MASK”
string in a prompt $PT_{k}$ is completed with the $PER$ from the pairwise
dataset, and the completed sentence $PT_{k}\wedge PER$ is fed to NER models to
obtain the confidence score of the $PER$ entity in $PT_{k}$.

###  3.3 Memorization Detection

Let $M_{ner}$ denote a NER model that has been fine-tuned on a pre-trained MLM
with dataset $D_{t}$. Such models can achieve high classification accuracy
(shown in Table 4 in the Appendix A.2) in predicting entity labels like $PER$
(i.e., B-PER and I-PER labels). To compute the confidence score of a $PER$, we
first gather the highest likelihood scores between B-PER and I-PER labels for
all name tokens of the $PER$, and then take the mean value of likelihood
scores as the final confidence for the $PER$. The confidence of a $PER$ in
$PT_{k}$ is defined as the notation $C_{k}(PER)$ (more details show in
Algorithm 1 in the Appendix A.1). After getting the confidence score for the
In-train and Out-train $PER$s in a name pair, we identify the sample
memorization of this $PER$ pair using the following definition:

######  Definition 1.

Sample Memorization (S-MEM): Given an In-train and Out-train PER pair, the
pair exposes the sample memorization (S-MEM) of a NER model if the In-train
PER gets a higher model confidence score than the Out-train PER.

The sample memorization of a $PER$ pair in the prompt $PT_{k}$ can be
formulated as:

$\textit{S-MEM}_{M_{ner}}(PT_{k}\wedge P^{In}_{i},PT_{k}\wedge
P^{Out}_{j})=\begin{cases}1&\text{if }C_{k}(P^{In}_{i})>C_{k}(P^{Out}_{j})\\\
0&\text{if }C_{k}(P^{Out}_{j})>C_{k}(P^{In}_{i})\end{cases}$

To detect the memorization of a NER model, we further define:

######  Definition 2.

Model Memorization (M-MEM): The memorization of a NER model is quantified by
the percentage of name pairs that expose sample memorization in a pairwise
dataset.

Finally, we formulate the performance of a prompt $PT_{k}$ on detecting
memorization of $M_{ner}$ as M-MEM scores:
$\textit{M-MEM}_{M_{ner}}(PT_{k})=\sum_{i=1}^{n}\sum_{j=1}^{n}\textit{S-MEM}_{M_{ner}}(PT_{k}\wedge
P^{In}_{i},PT_{k}\wedge P^{Out}_{j})/n^{2}$.

##  4 Experimental Setup

We performed our model memorization analysis on publicly accessible MLM-based
NER models fine-tuned by different developers on the CoNLL-2003 dataset (Tjong
Kim Sang & De Meulder, 2003). To provide a more comprehensive analysis, we
selected 6 models built on MLMs across 3 different pretraining schemes
(ALBERT-v2 (Lan et al., 2020), BERT (Devlin et al., 2019), and RoBERTa (Liu et
al., 2019)) and 2 model sizes (base and large sizes, -B and -L notations are
used correspondingly) for each scheme. More details about the models and
accessibility information can be found in Appendix A.2.

###  4.1 Strategies of Using the Prompt Set

We investigate three strategies for exploiting the prompt set, aiming for a
more thorough exploration of the impact of prompts on the memorization
detection of NER models.

Baselines (BS). $\emptyset$-PT uses the $PER$ without any additional text as
the input to query corresponding confidences for memorization detection. One-
PT and Mix-PT use the same strategies in Ali et al. (2022), One-PT employs one
hand-written prompt (“My name is XX.”), corresponding to the _known_ -setting
in Ali et al. (2022), while Mix-PT randomly chooses for each name one of the 5
hand-written prompts (“My name is XX.”, “I am XX.”, “I am named XX.”, “Here is
my name: XX.”, “Call me XX.”), like the _unknown_ -setting in Ali et al.
(2022).

Original Prompt (OPT) selects the best prompt (B-PT) and worst prompt (W-PT)
from the prompt set that achieved the highest and lowest M-MEM scores on the
dev set of $D_{pw}$.

Prompt Engineering (PTE) is inspired by Feng et al. (2018) which applies a
token-removal operation to the originally generated prompts. The goal is to
maximize the M-MEM score gap of the original best and worst prompts.
Specifically, the most important token (which contributes the most to score
improvement) and the least important token are removed from the worst and best
prompts, respectively, resulting in two modified prompts. This process is
iteratively repeated for the modified prompts until only one token (excluding
the PER) remains. The two modified prompts that achieved the highest and
lowest M-MEM scores on the dev set among all the modified prompts are named
BM-PT (best-modified prompt) and WM-PT (worst-modified prompt). It is
important to note that such prompts may be ungrammatical. More details are
presented in Section 5.5.

Ensembling of Prompts (EPT) employs several ensemble techniques: (1) Majority
Voting (MV) decide the S-MEM of a name pair if more than half of the prompts
in the set vote “1”; (2) Average Confidence Score (AVG-C) uses the average
$PER$ confidence over all prompts ($avg(\\{C_{k}(PER)\\}_{k=1}^{k=m}$)) as the
final confidence score of the $PER$; (3) Weighted Confidence Score (WED-C)
weights the $PER$ confidence of each prompt by its M-MEM score; (4) Maximum
confidence score (MAX-C) denote the maximum confidence of the $PER$ over all
prompts ($max(\\{C_{k}(PER)\\}_{k=1}^{k=m}$)) as the final confidence score;
(5) Minimum confidence score (MIN-C) is the contrast to MAX-C, which uses the
minimum value ($min(\\{C_{k}(PER)\\}_{k=1}^{k=m}$)) as the final confidence.

| Prompts | dev | test
---|---|---|---
| Rank | M-MEM | Rank | M-MEM
ALBERT-B | Bravo, MASK, what an impressive performance! | 1 | 74.84 | 2 | 75.10
Are you going to MASK’s art gallery opening tonight? | -1 | 58.92 | -1 | 58.74
ALBERT-L | Oh, MASK, you’re a true gem in our team. | 1 | 73.12 | 3 | 71.38
MASK, practice forgiveness towards yourself and others. | -1 | 64.60 | -3 | 64.70
BERT-B | Are you going to MASK’s art gallery opening tonight? | 1 | 73.96 | 1 | 71.84
Did MASK give you any advice on starting something new? | -1 | 65.11 | -2 | 63.30
BERT-L | What project is MASK working on? | 1 | 71.56 | 2 | 69.30
I had a chance to meet MASK’s family. | -1 | 67.78 | -1 | 64.42
RoBERTa-B | MASK, can you recommend a good restaurant in town? | 1 | 70.76 | 2 | 70.78
I had a great conversation with MASK at the party. | -1 | 57.00 | -2 | 59.63
RoBERTa-L | MASK, invest in meaningful relationships. | 1 | 75.27 | 1 | 72.79
MASK, practice playing the guitar. | -1 | 70.61 | -96 | 70.05
Table 2: The M-MEM scores and corresponding ranks of the best (Rank=1) and
worst (Rank=-1) prompts from the dev set on the test set.

##  5 Results and Analysis

Table 2 shows each model’s highest and lowest M-MEM scores achieved on the dev
set by the best and worst prompts in the original prompt set respectively. The
M-MEM scores vary considerably depending on the prompt: the gap between the
best and the worst prompt for a given model is as large as 16.40 percentage
points for ALBERT-B on the test set. We validate the statistical significance
of the results through Cochran’s $\mathcal{Q}$ test. We find that the
differences are significant for all models ($\text{p}<0.001$). The following
sections summarize and analyze the results to answer 6 research questions.

###  5.1 Do Model Properties Affect the Memorization Scores?

The 6 investigated models mainly cover 3 different pretraining schemes
(ALBERT-v2, BERT and RoBERTa) and 2 different model sizes for each scheme.
There is no pretraining scheme led to higher/lower memorization from results
in Table 2 and Figure 2 (a). However, we observe that the score gaps of large
models are generally smaller than those of base models in Table 2, this trend
does not apply to the standard deviations of the M-MEM scores of all prompts,
as shown by Figure 2 (a). For example, a similar standard deviation is
observed on ALBERT-B and ALBERT-L. In addition, the mean values of the M-MEM
scores of all prompts for the base and large models also indicate that model
sizes are not correlated with the memorization scores.

###  5.2 Are Prompt Performances Generalizable?

To examine how prompt performance generalizes across models and $PER$ pairs,
we compute the correlations (Kendall’s $\tau$ coefficient) among the M-MEM
scores of all the prompts for different models and data splits (Figure 2 (b)).
Generalizability across models: We observe weak correlations between prompts’
M-MEM scores for different models, even when comparing different variants of
the same model on the same data split. For example, M-MEM scores (dev set) for
BERT-B and BERT-L are negatively correlated. From this, we conclude that the
M-MEM scores of prompts do not generalize across models—the best prompts for a
given model may not be the best for other models. Generalizability across name
sets: Conversely, we found higher correlations between prompts’ M-MEM scores
across data splits when using the same model (diagonal of the right plot in
Figure 2). We conclude that the M-MEM scores of prompts generalize to some
extent across splits—for a given model, the best prompts for a given set of
$PER$ are likely to work well for a different set. Table 2 illustrates this by
comparing each model’s best and worst prompts, showing that the (model-
dependent) best and worst prompts for the dev set are still among the best and
worst for the test set. These results show that while prompt quality depends
on the model, it still generalizes for different $PER$.

(a)

(b)

Figure 2: (a): Mean and standard deviation of M-MEM scores of all prompts
across models on the dev set. (b): Correlations (Kendall’s $\tau$) of the
M-MEM scores of prompts across models. Left: correlations for the dev set
scores. Middle: correlations for the test set scores. Right: correlations
between dev and test set scores.

###  5.3 What is the Best Strategy to Use the Prompt Set?

Table 3 shows the results of applying different strategies to the prompt set.
The prompt set outperforms baselines: All the M-MEM scores achieved by best
prompts (B-PT) of the prompt set are higher than the two baselines in the dev
set (only one exception in the test set), showing the necessity to detect the
model memorization with diverse prompts. Prompt engineering outperforms the
prompt set: We observe that the best/worst-modified prompts (BM-PT and WM-PT)
using prompt engineering increase the score gap between the B-PT and the W-PT.
The BM-PT of RoBERTa-B improves by approximately 2 percentage points up on the
B-PT on the dev set, and the WM-PT decreases the M-MEM score of BERT-L 19
percentage points. Ensembling techniques do not bring improvement: The
ensemble techniques (EPT) do not improve/decrease the M-MEM scores from the
B-PT/W-PT scores.

|  | ALBERT-B | ALBERT-L | BERT-B | BERT-L | RoBERTa-B | RoBERTa-L
---|---|---|---|---|---|---|---
|  | dev | test | dev | test | dev | test | dev | test | dev | test | dev | test
BS |  $\emptyset$-PT | 71.11 | 72.44 | 70.81 | 68.91 | 66.78 | 66.10 | 70.16 | 68.11 | 66.39 | 65.46 | 74.40 | 70.29
One-PT | 70.42 | 71.18 | 69.02 | 67.57 | 72.15 | 70.31 | 69.91 | 66.49 | 68.57 | 70.72 | 72.68 | 70.22
Mix-PT | 68.63 | 70.02 | 67.58 | 66.98 | 70.59 | 69.27 | 69.29 | 66.56 | 66.47 | 67.29 | 72.80 | 70.42
OPT | B-PT | 74.84 | 75.10 | 73.12 | 71.38 | 73.96 | 71.84 | 71.56 | 69.30 | 70.76 | 70.78 | 75.27 | 72.79
W-PT | 58.92 | 58.74 | 64.60 | 64.70 | 65.11 | 63.30 | 67.78 | 64.42 | 57.00 | 59.63 | 70.61 | 70.05
PTE | BM-PT | 75.14 | 74.96 | 73.14 | 71.44 | 75.14 | 73.31 | 73.03 | 72.18 | 72.68 | 71.59 | 76.28 | 73.33
WM-PT | 41.92 | 44.53 | 63.76 | 63.52 | 59.15 | 62.04 | 48.68 | 49.98 | 55.29 | 58.21 | 69.15 | 68.52
EPT | MV | 73.24 | 74.3 | 70.96 | 69.71 | 70.75 | 69.34 | 69.91 | 67.98 | 64.87 | 66.30 | 73.73 | 71.05
AVG-C | 72.81 | 73.22 | 68.21 | 67.43 | 69.66 | 67.98 | 69.92 | 67.95 | 67.76 | 67.67 | 73.97 | 71.16
WED-C | 72.82 | 73.22 | 68.23 | 67.44 | 69.67 | 67.99 | 69.92 | 67.95 | 67.76 | 67.66 | 73.97 | 71.16
MAX-C | 71.53 | 73.71 | 70.65 | 68.99 | 73.52 | 71.40 | 69.97 | 67.31 | 61.47 | 63.22 | 73.69 | 71.25
MIN-C | 60.12 | 59.77 | 64.74 | 65.39 | 68.79 | 66.56 | 68.62 | 67.94 | 68.35 | 68.08 | 73.08 | 71.58
Table 3: Results of using different strategies on the prompts. One-PT and Mix-
PT are baselines adapted from Ali et al. (2022). Bold and underline highlight
the highest and lowest scores respectively.

###  5.4 Sentence Level Analysis: What Prompt Properties Impact the
Performance?

This section investigates different prompt properties that may influence the
M-MEM scores. Prompt type: Figure 3 (a) presents prompts’ M-MEM scores grouped
by prompt type. While some models show similar M-MEM scores across sentence
types, others are more sensitive to particular types. For example, RoBERTa-B
has very distinct M-MEM scores for different types: imperative prompts
generally yield higher scores than declarative prompts. Conversely, BERT-L has
similar M-MEM scores profiles for different prompt types. Person’s name token
position: Figure 3 (b) shows prompts’ M-MEM scores grouped by the token
position of the name. The closer the name is to the beginning of the prompt,
the higher the M-MEM score tends to be for RoBERTa-B and BERT-B. Prompt token
length: Figure 3 (c) contrasts prompts’ M-MEM scores and length (in number of
tokens). We observe a weak correlation between M-MEM scores and prompt length
for most models, meaning that longer prompts are generally more effective at
identifying memorization. However, BERT-B stands out as an outlier, showing a
moderate negative correlation existed between M-MEM scores and prompt token
lengths.

(a) Prompt types (declarative, exclamatory, imperative, and interrogative)

(b) Token positions of person’s name in the prompt

(c) Prompt token lengths

Figure 3: M-MEM scores grouped by different prompt properties on the dev set.

###  5.5 Token Level Analysis: How Tokens within a Prompt Impact the
Performance?

Figure 4 shows a token-level analysis of how tokens within a prompt impact the
M-MEM scores on the example of the BERT-B model111For the analysis of other
models considered in our experiments, please refer to Appendix A.3..
Individual token importance in both the best- and worst-performing prompts
(including modifications produced by token-removal) is calculated as the
difference between the M-MEM scores of the prompt with and without this token
on the dev set. These scores are then normalized using a softmax function. The
“MASK” token, acting as a placeholder for the $PER$, does not have an
importance score.

Removing the least important tokens from the prompt increases the M-MEM score,
while removing the most important tokens has the opposite effect. Iteratively
removing the least important tokens from the best-performing prompt (“Are you
going to MASKS’s art gallery opening tonight?”) enables an increase in the
M-MEM score up to 75.14 for the prompt “you going MASK’s art gallery”.
Continuing to remove tokens results in a slight decrease in prompt
performance; however, it still outperforms the initial prompt up to the noun-
phrase prompt “MASKS’s gallery” (which performs worse than the initial
prompt). In the case of the worst-performing prompt, removing the most
important token give from the initial prompt results in a major decrease in
the M-MEM score by approximately 3.5 percentage points. Further removal of
important tokens slightly reduces the score to a minimum of 59.15 for the
prompt “MASK you something ?”. We observe that almost all prompts, including
the ones produced by the removal of the tokens from the initial one and the
initial one itself, perform worse than the prompt containing only the $PER$
name (refer to the last line in Figure 4). The best-performing prompt consists
of only one token ”something”, apart from $PER$, achieving an M-MEM score of
67.69.

Figure 4: Analysis of the token importance in prompts and the M-MEM scores for
the BERT-B model. The best-performing (left) and the worst-performing (right)
prompts were selected on the dev set. The heatmaps are generated with leave-
one-token-out: at each step, the least important token is removed from the
best-performing prompt, and the most important token is removed from the
worst-performing prompt. The normalized token importance scores are under the
corresponding tokens. The removed tokens are underlined.

The tokens that function as verbs and the tokens that are located near the
$PER$ token tend to have a higher importance score. Tokens like “going”,
“give”, “practice”, “working”, “recommend”, etc. (see Figure 4 and Figures
6-10 in Appendix A.3) have a higher influence on the M-MEM score in each
considered prompt. A similar trend can be seen for the tokens “’s”, “any”,
“Oh,”, “Bravo,”, “What,” and “,” located nearer to the MASK token (substituted
by the $PER$ entity in the experiments) than tokens located further away.

The punctuation mark at the end of each prompt has minimal influence on the
prompt performance. In all our experiments, the token representing the final
punctuation mark consistently demonstrates a low importance score. On the
contrary, the punctuation marks in the middle of the prompt, particularly
those surrounding the MASK token (see, for example, Figure 6: “Bravo, MASK,
…”, and Figure 7 “Oh, MASK, …” in Appendix A.3) exhibit greater importance.
This can be explained by their role in altering sentence structure and
emphasizing distinct elements within it.

###  5.6 Self-attention Analysis: How Models Behave to Different Prompts and
PER?

An essential component of the transformer’s architecture is the multi-head
self-attention mechanism (Vaswani et al., 2017). The self-attention weights of
a model can show us where the model attends in the sequence and provide
insights into what affected the predictions. Figure 5 shows the attention
heatmaps of BERT-B for the best and the worst prompts with each prompt
completed with In-train and Out-train $PER$s separately. Specifically, we
first averaged the extracted attention weights of the tokens that correspond
to the $PER$ (“MASK”) for all completed sentences in a group (e.g., the best
prompt completed separately with all In-train $PER$s). Then we create a
heatmap using the averaged values over the attention heads and the layers.
Generally, we notice that attention tends to be distributed similarly given a
prompt, focusing on the first and the $PER$ tokens. This observation applies
to all analyzed models (heatmaps for the other 5 models are shown in Figure 11
in Appendix A.4).

Name level analysis: We observe a slightly higher focus on In-train $PER$s
than Out-train $PER$s when comparing the averaged attention weights on both
the best and worst prompts. Prompt level analysis: When observing the heatmaps
in the prompt level, we notice that models tend to focus more (or equally for
BERT-B shown in Figure 5) on the $PER$ tokens in the best prompt than the
worst, except when the $PER$ tokens in the worst prompt are positioned in the
beginning the prompt.

Figure 5: Attention heatmaps of the best and the worst prompts of BERT-B on
the dev set averaged over all attention heads and layers. The attention
weights corresponding to each prompt’s “MASK” token are used and averaged over
the In-train and Out-train $PER$s separately.

##  6 Conclusion

We studied memorization in fine-tuned MLM-based NER models. In contrast to
memorization in pre-training or auto-regressive language models, entities
(person’s name) occur only on the input side of the fine-tuning data, and not
on the output (which would be the PER tags), and therefore cannot be detected
with generation prompts as in previous works (Carlini et al., 2021; 2023). For
measuring memorization in this setting, we compared a fixed set of 5 hand-
written prompts to automatically generated 400 prompts, and we measured how
well those prompts could distinguish entities that occurred in the fine-tuning
data from those that did not, using a large set of candidate name entities
from Wikidata.

Detailed analysis showed that a prompt’s ability to detect memorization varies
between models, but is stable between different entity sets. We find a large
variability in the performance of generated prompts, and that many generated
prompts significantly outperform hand-written ones. The effectiveness of
prompts can be increased by removing the least important tokens even if the
prompts become ungrammatical after the removal.

##  7 Acknowledgements

We thank Vasisht Duddu and N. Asokan who dedicated time and expertise to
provide valuable feedback on this paper. This research has been funded by the
Vienna Science and Technology Fund (WWTF) [10.47379/VRG19008] “Knowledge-
infused Deep Learning for Natural Language Processing”.

## References

  * Ali et al. (2022) Rana Salal Ali, Benjamin Zi Hao Zhao, Hassan Jameel Asghar, Tham Nguyen, Ian David Wood, and Dali Kaafar.  Unintended Memorization and Timing Attacks in Named Entity Recognition Models, November 2022.  URL http://arxiv.org/abs/2211.02245.  arXiv:2211.02245 [cs].
  * Brown et al. (2021) Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar.  When is memorization of irrelevant training data necessary for high-accuracy learning?  In _Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing_ , pp. 123–132, 2021.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks, 2019.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pp. 2633–2650, 2021.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models, 2023.
  * Darji et al. (2023) Harshil Darji, Jelena Mitrović, and Michael Granitzer.  German bert model for legal named entity recognition.  _arXiv preprint arXiv:2303.05388_ , 2023.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Feldman (2020) Vitaly Feldman.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , pp. 954–959, 2020.
  * Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber.  Pathologies of neural models make interpretations difficult.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 3719–3728, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1407.  URL https://aclanthology.org/D18-1407.
  * Feng et al. (2024) Zijian Feng, Hanzhang Zhou, ZIXIAO ZHU, Junlang Qian, and Kezhi Mao.  Unveiling and manipulating prompt influence in large language models.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=ap1ByuwQrX.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2020.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Magar & Schwartz (2022) Inbal Magar and Roy Schwartz.  Data contamination: From memorization to exploitation.  _arXiv preprint arXiv:2203.08242_ , 2022.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David K Evans, and Taylor Berg-Kirkpatrick.  An empirical analysis of memorization in fine-tuned autoregressive language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 1816–1826, 2022.
  * Neel & Chang (2023) Seth Neel and Peter Chang.  Privacy issues in large language models: A survey.  _arXiv preprint arXiv:2312.06717_ , 2023.
  * OpenAI (2023) OpenAI.  Chatgpt conversation, December 2023.  Accessed from OpenAI’s ChatGPT.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L. Logan IV au2, Matt Gardner, and Sameer Singh.  Impact of pretraining term frequencies on few-shot reasoning, 2022.
  * Richie et al. (2023) Russell Richie, Victor M Ruiz, Sifei Han, Lingyun Shi, and Fuchiang Tsui.  Extracting social determinants of health events with transformer-based multitask, multilabel named entity recognition.  _Journal of the American Medical Informatics Association_ , pp. ocad046, 2023.
  * Sclar et al. (2023) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2023.
  * Tirumala et al. (2022) Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _Advances in Neural Information Processing Systems_ , 35:38274–38290, 2022.
  * Tjong Kim Sang & De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pp. 142–147, 2003.  URL https://www.aclweb.org/anthology/W03-0419.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, 2017.
  * Vrandečić & Krötzsch (2014) Denny Vrandečić and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85, 2014.
  * Wei et al. (2023) Chengwei Wei, Yun-Cheng Wang, Bin Wang, and C-C Jay Kuo.  An overview on language models: Recent developments and outlook.  _arXiv preprint arXiv:2303.05759_ , 2023.
  * Yu et al. (2023) Jianfei Yu, Ziyan Li, Jieming Wang, and Rui Xia.  Grounded multimodal named entity recognition on social media.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 9141–9154, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.508.  URL https://aclanthology.org/2023.acl-long.508.

##  Appendix A Appendix

###  A.1 Confidence of PER in a Prompt

We describe the details of obtaining the confidence of a $PER$ in a prompt in
Algorithm 1.

Algorithm 1 Confidence of $PER$ in a Prompt

Input: a person’s name $PER$, a prompt $PT_{k}$, a NER model $M_{ner}$,
tokenized $PER$ set $T_{PER}$, token set $T_{PT_{k}\wedge PER}$ after
tokenizing $PT_{k}\wedge PER$.

1:$C_{k}(PER)\leftarrow 0$

2:$Pr(T^{1}),Pr(T^{2}),...,Pr(T^{|T_{PT_{k}\wedge PER}|})\leftarrow
M_{ner}(T_{PT_{k}\wedge PER})$.

3:for $t\leftarrow 1$ to $|T_{PT_{k}\wedge PER}|$ do

4: if $T^{t}\in T_{PER}$ then

5: $C_{k}(T^{t})\leftarrow
max(Pr(\textit{B-PER}|T^{t}),Pr(\textit{I-PER}|T^{t}))$

6: $C_{k}(PER)\leftarrow C_{k}(PER)+C_{k}(T^{t})$

7:$C_{k}(PER)\leftarrow C_{k}(PER)/|T_{PER}|$.

8:Output:  $C_{k}(PER)$, confidence score of $PER$ for $PT_{k}$

###  A.2 MLM-based NER Models

We summarize the model performance details of the 6 explored fine-tuned NER
models based on MLMs in Table 4, and more details about the models and
accessibility information are described in the following.

ALBERT-B 222https://huggingface.co/ArBert/albert-base-v2-finetuned-ner and
ALBERT-L 333https://huggingface.co/Gladiator/albert-large-v2_ner_conll2003 NER
models are fine-tuned models of base and large versions of ALBERT-v2 (Lan et
al., 2020) respectively. ALBERT-v2 is a transformer-based language model that
is pre-trained on a substantial corpus of English text data through self-
supervised learning. This pretraining process entails exposure to raw textual
data without human annotation, thereby leveraging publicly available data
sources. Specifically, ALBERT-v2 utilizes two pretraining objectives: Masked
Language Modeling (MLM) and Sentence Ordering Prediction (SOP). Through MLM,
the model learns to predict masked tokens within a sequence of text, while SOP
tasks involve predicting the correct order of sentences within a document.
Notably, ALBERT-v2 aims to optimize efficiency and scalability by parameter
sharing across layers and employing parameter reduction techniques, such as
factorized embedding parameterization and cross-layer parameter sharing.

BERT-B 444https://huggingface.co/dslim/bert-base-NER and BERT-L
555https://huggingface.co/dslim/bert-large-NER NER models have the most
downloaded records on the public model platform. These two models are fine-
tuned models of BERT-base-cased and BERT-large-cased models (Devlin et al.,
2019). BERT is a transformer-based language model renowned for its ability to
capture bidirectional contextual information from text data. Similar to
ALBERT, BERT models are pre-trained on a large corpus of English data in a
self-supervised fashion, aiming to learn deep contextualized representations
of words or tokens. BERT employs the MLM objective, where it learns to predict
masked tokens within a sequence, and the Next Sentence Prediction (NSP)
objective, where it predicts whether two sentences are consecutive in the
original text. BERT’s architecture comprises transformer encoders stacked on
top of each other, allowing it to effectively capture contextual information
through attention mechanisms.

RoBERTa-B 666https://huggingface.co/dominiqueblok/roberta-base-finetuned-ner
and RoBERTa-L 777https://huggingface.co/Gladiator/roberta-large_ner_conll2003
NER models are fine-tuned models of RoBERTa-base and RoBERTa-large (Liu et
al., 2019). RoBERTa is another variant of the BERT model designed to improve
upon its pretraining methodology and performance. Like BERT and ALBERT-v2,
RoBERTa utilizes transformer-based architectures for language modeling tasks.
However, RoBERTa introduces several enhancements to the pretraining process,
including dynamic masking strategies, larger training datasets, and longer
training times. Notably, RoBERTa replaces the NSP objective with a more
extensive masking scheme during pretraining to improve the model’s robustness
and effectiveness in capturing contextual information from text data.
Additionally, RoBERTa employs larger batch sizes and longer training
sequences, contributing to its improved performance on downstream natural
language processing tasks.

Model | #Param. | P | R | F1 | ACC | Training epoch
---|---|---|---|---|---|---
ALBERT-B | 11.1M | 93.0 | 93.8 | 93.4 | 98.5 | 3
ALBERT-L | 16.6M | 94.0 | 94.5 | 94.2 | 98.7 | 5
BERT-B | 108M | 92.1 | 93.1 | 92.6 | 91.2 | -
BERT-L | 334M | 92.0 | 91.9 | 92.0 | 90.3 | -
Roberta-B | 124M | 95.3 | 96.0 | 95.7 | 98.9 | 3
Roberta-L | 354M | 96.2 | 96.9 | 96.6 | 99.4 | 5
Table 4: Self-reported performance of the NER Models on CoNLL-2003 evaluation
dataset.

###  A.3 Token-level Analysis

Same as Section 5.5, we provide token-level analysis for the rest of the 5
models in Figure 6-10.

Figure 6: Analysis of the token importance in prompts and the M-MEM scores for
the ALBERT-B model. The best-performing (on the left) and the worst-performing
(on the right) prompts were selected on the dev set. The heatmaps are
generated with leave-one-token-out: at each step, the least important token is
removed from the best-performing prompt (left), and the most important token
is removed from the worst-performing prompt (right). The removed tokens are
underlined.

Figure 7: Analysis of the token importance in prompts and the M-MEM scores for
the ALBERT-L model. The best-performing (on the left) and the worst-performing
(on the right) prompts were selected on the dev set. The heatmaps are
generated with leave-one-token-out: at each step, the least important token is
removed from the best-performing prompt (left), and the most important token
is removed from the worst-performing prompt (right). The removed tokens are
underlined.

Figure 8: Analysis of the token importance in prompts and the M-MEM scores for
the M-MEM score of the BERT-L model. The best-performing (on the left) and the
worst-performing (on the right) prompts were selected on the dev set. The
heatmaps are generated with leave-one-token-out: at each step, the least
important token is removed from the best-performing prompt (left), and the
most important token is removed from the worst-performing prompt (right). The
removed tokens are underlined.

Figure 9: Analysis of the token importance in prompts and the M-MEM scores for
the M-MEM score of the RoBERTa-B model. The best-performing (on the left) and
the worst-performing (on the right) prompts were selected on the dev set. The
heatmaps are generated with leave-one-token-out: at each step, the least
important token is removed from the best-performing prompt (left), and the
most important token is removed from the worst-performing prompt (right). The
removed tokens are underlined.

Figure 10: Analysis of the token importance in prompts and the M-MEM scores
for the M-MEM score of the RoBERTa-L model. The best-performing (on the left)
and the worst-performing (on the right) prompts were selected on the dev set.
The heatmaps are generated with leave-one-token-out: at each step, the least
important token is removed from the best-performing prompt (left), and the
most important token is removed from the worst-performing prompt (right). The
removed tokens are underlined.

###  A.4 Self-attention Analysis

Same as Section 5.6, we provide self-attention analysis for the rest of the 5
models in Figure 11.

(a) ALBERT-B

(b) ALBERT-L

(c) BERT-L

(d) RoBERTa-L

(e) RoBERTa-L

Figure 11: Attention heatmaps of the best and the worst prompts across 5
models on the dev set averaged over all attention heads and layers. The
attention weights corresponding to each prompt’s “MASK” token are used and
averaged over the In-train and Out-train $PER$ sets separately.