# PULSAR: Graph based Positive Unlabeled Learning with Multi Stream Adaptive
Convolutions for Parkinson’s Disease Recognition

Md. Zarif Ul Alam 1, Md Saiful Islam 2, Ehsan Hoque 2, M Saifur Rahman 1

###### Abstract

Parkinson’s disease (PD) is a neuro-degenerative disorder that affects
movement, speech, and coordination. Timely diagnosis and treatment can improve
the quality of life for PD patients. However, access to clinical diagnosis is
limited in low and middle income countries (LMICs). Therefore, development of
automated screening tools for PD can have a huge social impact, particularly
in the public health sector. In this paper, we present PULSAR, a novel method
to screen for PD from webcam-recorded videos of the finger-tapping task from
the Movement Disorder Society - Unified Parkinson’s Disease Rating Scale (MDS-
UPDRS). PULSAR is trained and evaluated on data collected from 382
participants (183 self-reported as PD patients). We used an adaptive graph
convolutional neural network to dynamically learn the spatio temporal graph
edges specific to the finger-tapping task. We enhanced this idea with a multi
stream adaptive convolution model to learn features from different modalities
of data critical to detect PD, such as relative location of the finger joints,
velocity and acceleration of tapping. As the labels of the videos are self-
reported, there could be cases of undiagnosed PD in the non-PD labeled
samples. We leveraged the idea of Positive Unlabeled (PU) Learning that does
not need labeled negative data. Our experiments show clear benefit of modeling
the problem in this way. PULSAR achieved 80.95% accuracy in validation set and
a mean accuracy of 71.29% (2.49% standard deviation) in independent test,
despite being trained with limited amount of data. This is specially promising
as labeled data is scarce in health care sector. We hope PULSAR will make PD
screening more accessible to everyone. The proposed techniques could be
extended for assessment of other movement disorders, such as ataxia, and
Huntington’s disease.

Figure 1: Overview of the PD screening pipeline. A participant can perform
finger-tapping task in front of a computer webcam. A hand tracking model is
used to locate the key points of the hand. A spatio temporal graph is
constructed specifically for the finger-tapping task. Four different feature
streams (joint, bone, velocity and acceleration) are generated and fed to the
proposed PULSAR model for prediction.

## Introduction

Parkinson’s disease (PD) is one of the common degenerative diseases of the
nervous system (Ji et al. 2018). It is characterized by a variety of life-
changing motor dysfunction symptoms, including tremor, Bradykinesia (slowness
of movement), rigidity (limb stiffness), impaired balance and gait, etc. Over
10 million people worldwide are affected by PD (Wikipedia contributors 2022).
In 2016, PD resulted in about 211,000 deaths globally, an increase of 161%
since 1990. The diagnosis of PD mainly relies on clinical criteria based on
the Parkinsonian symptoms (e.g., tremor, Bradykinesia), and medical history.
However, the clinical diagnostic is challenged by the subjective opinions or
experiences of different medical experts. In addition, it is not accessible to
many individuals since the number of neurologists is very limited in some
countries (Kissani et al. 2022). Therefore, an efficient and remote automatic
PD diagnosis system is valuable for supporting clinicians with more robust
diagnostic decision-making.

With the advancement in computer vision technology, sensing of subtle movement
of face and body has become possible, and this has the potential to lead to
important medical and physiological implications. However, healthcare data
(specially, data from individuals with PD) is hard to obtain and the small
size of the datasets limits application of state-of-the art machine learning
models. Here we introduce PULSAR – a graph neural network based deep learning
architecture that can be trained with limited data. In addition, in a dataset
that uses patient-reported PD diagnosis as the ground truth, “negative” data
samples remain unreliable – many people live with PD without being diagnosed.
In context of healthcare this is very common. Rather than expecting clean
data, we focused on dealing with the problem objectively. To the best of our
knowledge, we are the first to explore Positive Unlabeled (PU) learning in the
context of PD screening, which eliminates the need for reliable “negative”
samples. An overview of the PD screening pipeline leveraging PULSAR model is
shown in Figure 1. PULSAR has been tested through a comprehensive set of
experiments, the results of which support its efficacy. As many movement
disorders (e.g., ataxia, Huntington’s disease) share similarity in terms of
the tasks used for screening/diagnosis and data collection, our proposed
approach could be applied for those diseases, potentially improving access to
neurological care. In this regard, the data processing and model training
codes and the processed de-identified dataset will be made publicly available
upon acceptance of the paper.

In summary, we make the following contributions:

  * •

We propose a novel multi stream adaptive convolution model which allows PD
screening from videos of the finger-tapping task with limited data.

  * •

We propose PU learning for PD screening that eliminates the necessity of
labeled negative data for model training.

  * •

The model achieved 80.95% and 71.29% accuracy on the validation and test set,
respectively, demonstrating the efficacy of our proposed approach.

## Related Work

Clinical diagnosis of PD involves a comprehensive assessment of a patient’s
medical history, neurological examination, and the presence of characteristic
motor features. Numerous clinical studies focus on identifying motor symptoms,
analyzing brain imaging data, and investigating physiological biomarkers to
diagnose PD. For example, neuroimaging techniques have furthered the
understanding of PD pathophysiology. PET (Positron Emission Tomography) and
SPECT (Single Photon Emission Computed Tomography) scans have been used to
visualize dopamine transporter binding in the basal ganglia, aiding in
differential diagnosis and disease progression assessment (Marek et al. 1996).
These clinical approaches provide essential groundwork for understanding PD
and serve as a reference for developing computational methods. But the
clinical diagnosis method is not cost-effective and as such it is very hard to
make PD diagnosis accessible for all.

Machine Learning (ML) algorithms have recently been used for PD detection and
diagnosis tasks (Senturk 2020; Li and Li 2022; Govindu and Palwe 2023) due to
their powerful predictive performance. Studies have shown that ML can provide
early and reliable diagnosis of patients and help doctors make decisions by
analyzing speech, gait or motor tasks. In parallel to traditional ML
techniques, Deep Learning (DL) techniques have attracted a lot of attention
because of their powerful automatic feature extraction capabilities (Shahid
and Singh 2020; Srinidhi, Ciga, and Martel 2021). For example, Sivaranjini and
Sujatha developed a deep learning-based approach that classifies PD using
Magnetic Resonance (MR) images, and Johri, Tripathi et al. utilized neural
network-based techniques to detect PD from gait and speech, showcasing the
potential of DL methods. A common problem with many of these models is that
they do not make diagnosis easily available to everyone. For example,
Neuroimaging or MRI based models, despite performing well, are expensive and
also intrusive. Speech-based models have limitations too, as they are not
easily generalizable because the language and pronunciation habits of people
in different countries vary significantly. Also, in most cases, the training
set is too small that these models do not generalize well.

Our work is inspired by recent developments in Graph Neural Networks (GNNs).
GNN has emerged as a powerful tool for analyzing complex and structured data,
making it particularly valuable in the context of healthcare and medical data
analysis. Yan, Xiong, and Lin and Shi et al. achieved remarkable performance
in skeleton based action recognition using graph convolutional networks. Kim
et al. introduced a GNN-based approach for predicting disease progression in
Alzheimer’s patients using brain connectivity graphs based on longitudinal
neuroimaging data. Zhao et al. used an attention-based graph neural network
(AGNN) for early diagnosis of PD via diffusion tensor imaging (DTI) data and
phenotypic information. In addition, PU learning enables a classifier to learn
from positive and unlabeled samples (Qiu, Zhang, and Zhu 2009; Nguyen, Li, and
Ng 2011), which is particularly significant for medical data analysis, as the
absence of a disease is often costly to verify.

## Dataset

The dataset contains recorded videos from $382$ unique participants completing
the finger-tapping task in front of a computer webcam. The finger-tapping task
involves repeatedly tapping the thumb-finger with the index-finger (10 times)
as fast and as big as possible. The participants completed the task twice,
once with the right hand, and another with the left hand. Each participant was
asked to self-report whether they were clinically diagnosed with Parkinson’s
disease or not. Data was collected using the PARK tool (Langevin et al. 2019),
a public website111https://parktest.net accessible from computers with major
Internet browsers.

The training and validation dataset consists of recorded vidoes from $200$
unique participants. Out of them $100$ are self reported PD patients. 20% of
this set is used for validation purpose. The independent test set has videos
from $182$ unique participants different from the training and validation set.
Out of them $83$ are self reported PD patients. Each of the videos were
segmented into clips of 80 frames and each of this clips are treated as one
sample of our dataset. Videos from individuals with self-reported PD were
labeled as positive samples, while individuals not reporting a diagnosed PD
remained unlabeled. Table 1 presents demographic information of the study
participants. It is notable that, due to low prevalence of PD (in 2022, the
overall prevalence of PD among persons older than 45 was 0.572 per 100 (Willis
et al. 2022)), typically, the unlabeled cohort are considered to not have PD
(Islam et al. 2023). However, it is possible that some of the unlabeled
participants did have PD, but remained undiagnosed. This makes positive
unlabeled learning particularly suitable for this problem.

Characteristics | With PD | Unlabeled | Total
---|---|---|---
| No. of
---
Participants, n
183 | 199 | 382
Sex, n (%)
| Male | 112 (61.2%) | 77 (38.7%) | 189 (49.5%)
| Female | 69 (37.7%) | 122 (61.3%) | 191 (50.0%)
| Not available | 2 (1.1%) | 0 (0.0%) | 2 (0.5%)
Age in years, n (%)
| Below 20 | 1 (0.5%) | 4 (2.0%) | 5 (1.3%)
| 20-39 | 4 (2.2%) | 20 (10.0%) | 24 (6.3%)
| 40-59 | 48 (26.2%) | 47 (23.6%) | 95 (24.9%)
| 60-79 | 119 (65.0%) | 126 (63.3%) | 245 (64.1%)
| Above 80 | 9 (4.9%) | 1 (0.5%) | 10 (2.6%)
| Not available | 2 (1.1%) | 1 (0.5%) | 3 (0.8%)

Table 1: Demographic characteristics of the participants. With PD column
represents the participants who self-reported to be diagnosed with PD, while
the rest of the participants remained unlabeled.

## Methodology

Our developed method PULSAR focuses on the finger-tapping task. This is
commonly used in neurological exams to evaluate bradykinesia (i.e., slowing of
movement) in upper extremities, which is a key symptom of PD (Hughes et al.
1992). The input consists of video recordings of each participant sitting in
front of a webcam and doing the finger-tapping task.

#### Video Augmentation

In our experiments, we employed three types of video augmentation to increase
the size and diversity of our dataset. These are horizontal flip, vertical
flip, and horizontal-vertical flip of the input videos. These transformations
simulate variations in camera position and hand orientation and were applied
to each video in the dataset. By flipping the videos, the size of the dataset
became fourfold the original size. We also normalize the pixel intensities of
each frame and resized them to a fixed resolution to ensure that the model is
trained on consistent inputs.

#### Data Cleaning

Data cleaning was performed to prevent learning from irrelevant or noisy data.
We removed frames where no hand was visible – typically at the beginning and
end of the videos when the subject was preparing to start the task or lowering
the hand after completing the task (Figure 3). To this end, we used MediaPipe
Hands (Zhang et al. 2020) to detect the presence of a hand in each frame, and
removed the frames where no hand was visible. We were thus left with a clean
and consistent dataset containing only relevant frames. This step proved
crucial as the model subsequently became robust, with a considerable boost in
performance (11% increase in accuracy in validation set). Furthermore, this
expedited our training process because the number of frames got considerably
reduced.

Figure 2: Demonstration of the described data cleaning process.333Images are
for demonstration purpose and not part of the dataset.

#### Hand Key Point Extraction

We extracted hand key point coordinates from RGB video data using MediaPipe
that offers high accuracy and fast inference speed. Twenty one 3-D hand key
points were extracted, including four joints per finger and one wrist joint,
following the structure shown in Figure 3. The minimum detection confidence
and minimum tracking confidence were empirically set to 0.8 and 0.9
respectively to reduce false positives.

#### Spatio Temporal Graph Construction

The inherent graph structure connects each joint with its immediate natural
neighbor, culminating in a graph with 21 vertices and 20 edges. However, this
proposed graph might lack the capacity to capture fine-grained hand movements
(Li et al. 2019). For the purpose of obtaining more contextually significant
semantic information, we introduced three types of augmented edges (Figure
3a). The first type of edge connects the fingertips to the base of the
adjacent finger to the right for the right hand (to the left for the left
hand). The second type of edge connects the fingertip to the middle joint of
the same finger. The third type establishes a connection between the tip of
the thumb and the index finger. These augmented edges are able to capture
additional insights into various hand states, such as the horizontal and
vertical distance between two fingers and finger bending. The third type of
connection is particularly relevant for the finger-tapping task.

(a)

(b)

Figure 3: Spatio Temporal Graph Construction. (a) The spatial connections of
the finger and wrist joints. Natural joint connections are denoted by solid
lines. The first type of augmented edge is denoted by dashed lines, the second
type by dotted lines, and the third by a blue dashed arrowhead. (b) The
temporal connections are between the same joints in the successive frames.

#### Adaptive Graph Convolution

In the context of the spatio temporal graph convolution network, the graph
convolution process can be defined as follows (Yan, Xiong, and Lin 2018):

$\mathbf{f}_{out}=\sum_{k}^{K_{v}}\mathbf{W}_{k}\left(\mathbf{f}_{in}\mathbf{A}_{k}\right)\odot\mathbf{M}_{k}\vspace{-1mm}$
(1)

where $\mathbf{f}_{in}$ stands for the input feature map, $\mathbf{f}_{out}$
signifies the resulting feature map, $K_{v}$ represents the spatial
dimension’s kernel size, $\mathbf{W}_{k}$ is the weight vector of the $1\times
1$ convolution. $\mathbf{M}_{k}$ is an $N\times N$ attention map that
indicates the significance of each vertex. The symbol $\odot$ denotes the dot
product operation. The matrix $A_{k}$ determines the presence of connections
between two vertices. For the temporal dimension, where each vertex has a
fixed number of neighbors (corresponding to joints in consecutive frames), the
execution of graph convolution is akin to the traditional convolution
operation, following the approach described by Yan, Xiong, and Lin.

In a graph convolutional network, the convolution process involves aggregating
data from neighboring nodes within the graph. This combined data, along with
the node’s inherent characteristics, is subsequently passed through a non-
linear activation function to generate the convolutional output. But having a
fixed predefined graph might not be optimal for capturing nuanced distinctions
in motor tasks performed by Parkinson’s patients. Such an approach could limit
the capacity of the graph convolutional network (GCN) to acquire comprehensive
insights from alternative connections. The fixation of the graph structure
throughout all GCN layers contradicts the principle that the neural network
could acquire diverse features at each layer. Adaptive graph convolution as
presented in Shi et al., is a type of graph convolution that allows for
learnable weights on the aggregation step of the convolution operation. Unlike
traditional GCNs, where the aggregation weights used to combine data from
neighboring nodes remain fixed and consistent across all nodes, adaptive graph
convolution allows these weights to be learned during the training process.

The adaptive graph convolution layer optimizes the graph’s structure alongside
other network parameters through an end-to-end learning approach, as
demonstrated by Shi et al.. Notably, the graph’s configuration is distinct for
various layers and instances, significantly enhancing the model’s
adaptability. It is also designed as a residual component, ensuring the
stability of the underlying model.

To make the graph structure adaptive, Eq. 1 is changed as:
$\mathbf{f}_{\text{out
}}=\sum_{k}^{K_{v}}\mathbf{W}_{k}*\mathbf{f}_{in}\left(\mathbf{A}_{k}+\mathbf{B}_{k}+\mathbf{C}_{k}\right)$
(Shi et al. 2019). The adaptive spatial graph convolution layer integrates
both the provided adjacency matrix ($A_{k}$) and parameterized, optimized
adjacency matrices ($B_{k}$ and $C_{k}$). These matrices are detailed as
follows.

  * •

$\mathbf{A_{k}}$ denotes the original normalized adjacency matrix, capturing
inherent physical structure of human hands.

  * •

$\mathbf{B_{k}}$ represents a fully learnable and parameterized adjacency
matrix. The absence of constraints on the values of $B_{k}$ permits the graph
to be learned based on training data. $B_{k}$ is capable of generating novel
connections, similar to attention mechanism, yet offering greater flexibility
due to its ability to establish connections absent in the original physical
graph.

  * •

$\mathbf{C_{k}}$ signifies a data-dependent adjacency matrix, derived by
embedding input features via a $1\times 1$ convolutional operation and a
softmax function. To ascertain the presence and strength of connections
between vertex pairs, a normalized embedded Gaussian function computes the
similarity between the vertices (Shi et al. 2019).

Instead of directly substituting the original $A_{k}$ with $B_{k}$ or $C_{k}$,
they are incorporated alongside it. This strategy amplifies model flexibility
while preserving performance.

#### Positive Unlabelled Learning

Let random variables $\boldsymbol{x}\in\mathbb{R}^{d}$ and $y\in\\{+1,-1\\}$
be equipped with probability density $p(\boldsymbol{x},y)$, where $d$ is a
positive integer. In a binary classification problem from $\boldsymbol{x}$ to
$y$, let’s assume there are three sets of samples: positive $(\mathrm{P})$,
negative $(\mathrm{N})$, and unlabeled $(\mathrm{U})$.

$\displaystyle\mathcal{X}_{\mathrm{P}}:=\left\\{\boldsymbol{x}_{i}^{\mathrm{P}}\right\\}_{i=1}^{n_{\mathrm{P}}}\stackrel{{\scriptstyle\text{
i.i.d. }}}{{\sim}}p_{\mathrm{P}}(\boldsymbol{x}):=p(\boldsymbol{x}\mid y=+1),$
$\displaystyle\mathcal{X}_{\mathrm{N}}:=\left\\{\boldsymbol{x}_{i}^{\mathrm{N}}\right\\}_{i=1}^{n_{\mathrm{N}}}\stackrel{{\scriptstyle\text{
i.i.d. }}}{{\sim}}p_{\mathrm{N}}(\boldsymbol{x}):=p(\boldsymbol{x}\mid y=-1),$
$\displaystyle\mathcal{X}_{\mathrm{U}}:=\left\\{\boldsymbol{x}_{i}^{\mathrm{U}}\right\\}_{i=1}^{n_{\mathrm{U}}}\stackrel{{\scriptstyle\text{
i.i.d.
}}}{{\sim}}p(\boldsymbol{x}):=\theta_{\mathrm{P}}p_{\mathrm{P}}(\boldsymbol{x})+\theta_{\mathrm{N}}p_{\mathrm{N}}(\boldsymbol{x}),$

where $\theta_{\mathrm{P}}:=p(y=+1),\quad\theta_{\mathrm{N}}:=p(y=-1)$ are the
class-prior probabilities for the positive and negative classes such that
$\theta_{\mathrm{P}}+\theta_{\mathrm{N}}=1$.

Let $g:\mathbb{R}^{d}\rightarrow\mathbb{R}$ be an arbitrary real-valued
decision function for binary classification, and classification is performed
based on its sign. Let $\ell:\mathbb{R}\rightarrow\mathbb{R}$ be a loss
function such that $\ell(m)$ generally takes a small value for large margin
$m=yg(\boldsymbol{x})$. Let
$R_{\mathrm{P}}(g),R_{\mathrm{N}}(g),R_{\mathrm{U},\mathrm{P}}(g)$, and
$R_{\mathrm{U},\mathrm{N}}(g)$ be the risks of classifier $g$ under loss
$\ell$ :

$\begin{array}[]{rrr}R_{\mathrm{P}}(g):=\mathrm{E}_{\mathrm{P}}[\ell(g(\boldsymbol{x}))],&R_{\mathrm{N}}(g):=\mathrm{E}_{\mathrm{N}}[\ell(-g(\boldsymbol{x}))]\\\
R_{\mathrm{U},\mathrm{P}}(g):=\mathrm{E}_{\mathrm{U}}[\ell(g(\boldsymbol{x}))],&R_{\mathrm{U},\mathrm{N}}(g):=\mathrm{E}_{\mathrm{U}}[\ell(-g(\boldsymbol{x}))]\end{array}$

where $\mathrm{E}_{\mathrm{P}},\mathrm{E}_{\mathrm{N}}$, and
$\mathrm{E}_{\mathrm{U}}$ denote the expectations over
$p_{\mathrm{P}}(\boldsymbol{x}),p_{\mathrm{N}}(\boldsymbol{x})$, and
$p(\boldsymbol{x})$, respectively. Since we do not have any samples from
$p(\boldsymbol{x},y)$, the true risk $R(g)=$
$\mathrm{E}_{p(\boldsymbol{x},y)}[\ell(yg(\boldsymbol{x}))]$, which we want to
minimize, should be recovered without using $p(\boldsymbol{x},y)$ as shown
below.

When we have both positive and negative samples in a supervised classification
task (i.e., PN classification), the risk can be defined as

$\displaystyle R_{\mathrm{PN}}(g)$
$\displaystyle:=\theta_{\mathrm{P}}\mathrm{E}_{\mathrm{P}}[\ell(g(\boldsymbol{x}))]+\theta_{\mathrm{N}}\mathrm{E}_{\mathrm{N}}[\ell(-g(\boldsymbol{x}))]$
(2)
$\displaystyle=\theta_{\mathrm{P}}R_{\mathrm{P}}(g)+\theta_{\mathrm{N}}R_{\mathrm{N}}(g)$

which is equal to $R(g)$, but $p(\boldsymbol{x},y)$ is not included.

In PU classification, labeled data for the negative class is absent. Unlabeled
data originating from marginal density $p(\boldsymbol{x})$ can be utilized
instead. The objective here is to train a classifier using only positive and
unlabeled data. While the basic approach is to discriminate $\mathrm{P}$ and
$\mathrm{U}$ data (Elkan and Noto 2008), naively classifying $\mathrm{P}$ and
$\mathrm{U}$ data causes a bias. To address this problem, Du Plessis, Niu, and
Sugiyama proposed a risk equivalent to the $\mathrm{PN}$ risk but without
including $p_{\mathrm{N}}(\boldsymbol{x})$. The key idea is to utilize
unlabeled data to evaluate the risk for negative samples in the PN risk.
Replacing the second term in Eq. 2 with 444The equation comes from the
definition of the marginal density
$p(\boldsymbol{x})=\theta_{\mathrm{P}}p_{\mathrm{P}}(\boldsymbol{x})+\theta_{\mathrm{N}}p_{\mathrm{N}}(\boldsymbol{x})$

$\theta_{\mathrm{N}}\mathrm{E}_{\mathrm{N}}[\ell(-g(\boldsymbol{x}))]=\mathrm{E}_{\mathrm{U}}[\ell(-g(\boldsymbol{x}))]-\theta_{\mathrm{P}}\mathrm{E}_{\mathrm{P}}[\ell(-g(\boldsymbol{x}))]\text{,
}$

the risk in PU classification (the PU risk) is obtained as follows

$\displaystyle R_{\mathrm{PU}}(g)$
$\displaystyle:=\theta_{\mathrm{P}}\mathrm{E}_{\mathrm{P}}[\widetilde{\ell}(g(\boldsymbol{x}))]+\mathrm{E}_{\mathrm{U}}[\ell(-g(\boldsymbol{x}))]$
$\displaystyle=\theta_{\mathrm{P}}R_{\mathrm{P}}^{\mathrm{C}}(g)+R_{\mathrm{U},\mathrm{N}}(g),$

where
$R_{\mathrm{P}}^{\mathrm{C}}(g):=\mathrm{E}_{\mathrm{P}}[\widetilde{\ell}(g(\boldsymbol{x}))]$
and $\widetilde{\ell}(m)=\ell(m)-\ell(-m)$ is a composite loss function.

#### PULSAR

We propose PULSAR, a spatio temporal graph convolution model, where we used
the idea of Adaptive Graph Convolution (Shi et al. 2019) in context of hand
skeletons, allowing us to capture intricate temporal patterns and spatial
relationships within the hand movements. In addition to that, to overcome the
challenges inherent to the specific problem in hand, we frame the task of
detecting Parkinson’s Disease from the finger-tapping task as a Positive
Unlabeled (PU) Learning problem where we use a modified risk estimator.

In a standard spatio temporal graph convolution model (e.g., used for action
recognition), the feature vector attached to each vertex only contains the
coordinates of the joints, i.e., the first order information of the skeleton
data. The hand joint coordinates by itself can be utilized for the PD
detection model. However, the second order information, which represent the
feature of bones between two joints, or third order information like velocity
and acceleration, hasn’t been exploited. By leveraging bone vectors derived
from joint positions, we capture the relative orientations between joints,
which is crucial in assessing motor impairment. In the finger-tapping task,
the velocity and acceleration of the tapping motion can provide insights into
the patient’s motor control abilities. In a healthy individual, the tapping
motion is expected to exhibit relatively smooth and controlled movements,
resulting in consistent velocity and acceleration profiles. However, in
Parkinson’s patients, motor impairment can lead to irregular and jerky
movements, resulting in variations in velocity and acceleration patterns.

To utilize additional information, we integrate streamed attributes involving
secondary and tertiary data aspects such as bone-related details, velocity,
and acceleration (Figure 4). The bone constructs are established as vectors
denoting the directional link between two joints. This directional aspect is
preserved by subtracting joint positions. Velocity and acceleration
representations are computed by taking the first and second order derivative
of joint positions across frames respectively. To avoid mixing the information
about the different features, all four models are trained separately.

The model takes in an embedded sequence of joints along with its corresponding
adjacency matrix as the input. While the spatial convolution component remains
unchanged, the temporal graph convolutional part is executed following the
approach proposed by Yan, Xiong, and Lin. This involves applying a
$K_{t}\times 1$ convolution on the $C\times T\times V$ feature maps, where $C$
is the number of input features, $T$ is the input time steps, $V$ is the
number of vertices in the graph and $K_{t}$ is the kernel size of temporal
dimension. Both the spatial graph convolutional network (GCN) and the temporal
GCN are succeeded by a batch normalization (BN) layer and a Rectified Linear
Unit (ReLU) layer. Within each block, a dropout layer with a dropout rate of
0.5 is positioned between the spatial GCN and the temporal GCN. The network we
propose is fed by a four-dimensional matrix of shape $N\times C\times T\times
V$ where $N$ represents the batch size.

We used two AGCN blocks in our network with increasing output feature
dimensions. As a preprocessing step, we included a batch normalization layer
to standardize the input data. The output of the model is mapped to the
corresponding output classes through a global average pooling layer (GAP)
followed by a fully connected layer (FC). The AGCN-block is a fundamental
building block of the PULSAR model, which involves a spatial graph convolution
followed by a temporal graph convolution with specific kernel sizes. Each of
the AGCN blocks has a residual connection for stabilizing the training.

Figure 4: Feature representation of multi stream network of PULSAR

### Training Details

During the training phase, a batch size of 64 and 2 input features (x-y
coordinates) are set. The choice of batch size was done through an initial
experiment with mini-batch sizes of 32, 64, and 128, where the size of 64
yielded the highest validation accuracy. This decision was also guided by the
benefits of smaller batch sizes, which are known to yield smoother minimizers
and improved generalization capabilities, as highlighted by (Keskar et al.
2016). The temporal dimension $T$ was chosen to be 80 based on empirical
observations, and $V$ was set to 21, which is the number of available hand
joints. Adam optimizer was used with an initial learning rate of $1e^{-4}$. A
plateau learning rate scheduler was implemented, adjusting according to the
validation accuracy, with a reduction factor of 0.5 and a patience of 5
epochs. The specific learning rate and scheduler parameters were fine-tuned
across a sequence of experiments, varying the learning rates from $1e^{-3}$ to
$1e^{-5}$ and the reduction factors from 0.1 to 0.5. The models were trained
for a maximum of 30 epochs.

Name |  Description
---|---
JS |  Model based on joint Stream features
JS_PU |  Model based on joint Stream features with PU Learning
JS_AC |  Model based on joint Stream features with Adaptive Convolution
JS_AC_PU |  Model based on joint Stream features with Adaptive Convolution and PU Learning
PULSAR |  Model based on joint, bone, velocity and acceleration streams (i.e., multi stream) features with Adaptive Convolution and PU Learning

Table 2: Acronym and short description of baseline models along with PULSAR.

## Results

In this section we evaluate the performance of PULSAR and compare it with
several baseline models as mentioned in Table 2. All experiments were
conducted on PyTorch deep learning framework, 1 Nvidia GTX 1070 GPU and 1
Nvidia Tesla P100 GPU.

$Model$ | $Acc$ | $Prec$ | $Rec$ | $F_{1}^{m}$ | $F_{1}^{w}$ | $AUC$
---|---|---|---|---|---|---
JS | 60.88 | 61.10 | 61.09 | 60.87 | 60.86 | 61.08
JS_PU | 69.05 | 66.91 | 65.70 | 66.04 | 68.48 | 65.69
JS_AC | 65.90 | 66.05 | 66.06 | 65.90 | 65.91 | 66.06
JS_AC_PU | 70.24 | 68.32 | 68.02 | 68.16 | 70.12 | 68.02
PULSAR | 80.95 | 84.32 | 82.76 | 80.87 | 80.73 | 82.75

Table 3: Performance comparison of PULSAR with baseline models on the
validation set. Best result for each metric is shown in bold-face. $Acc$:
Accuracy, $Prec$: Precision, $Rec$: Recall, $F_{1}^{m}$: Macro F1,
$F_{1}^{w}$: Weighted F1, $AUC$: Area under Receiver Operating Characteristic
Curve.

### Performance on Validation Set

The performance comparison of PULSAR with baseline models on the validation
set is shown in Table 3. PULSAR has the highest accuracy, precision, recall,
F1 (macro as well as weighted) and area under receiver operating
characteristic Curve (AUROC). Compared to the joint stream based model, PULSAR
enjoys improvements of 33% and 35% in accuracy and AUROC respectively. On the
other hand, the inclusion of bone, velocity and acceleration streams alongside
joint stream (i.e., from JS_AC_PU model to PULSAR) improves the accuracy by
15%. The benefit of PU learning is also quite clear from the validation
results. From JS to JS_PU model, accuracy increased by 13%. On the other hand,
inclusion of adaptive convolution to the vanilla JS model increased accuracy
by 8%. Overall, all these added components – multiple streams, PU learning and
adaptive convolution – significantly improved PULSAR over the baseline model
with joint stream alone.

$Model$ | $Acc$ | $Prec$ | $Rec$ | $F_{1}^{m}$ | $F_{1}^{w}$ | $AUC$
---|---|---|---|---|---|---
JS | 63.46 | 64.20 | 63.56 | 62.92 | 63.05 | 63.56
JS_PU | 64.12 | 66.47 | 64.43 | 62.93 | 62.97 | 64.43
JS_AC | 66.68 | 66.57 | 66.64 | 66.54 | 66.71 | 66.64
JS_AC_PU | 68.41 | 69.70 | 68.61 | 67.89 | 67.97 | 68.61
PULSAR | 71.29 | 71.21 | 71.22 | 71.06 | 71.29 | 71.22

Table 4: Performance comparison of PULSAR with baseline models on the
independent test set. Best result for each metric is shown in bold-face. All
the values are averaged over 20 replicates.

### Performance on Independent Test Set

From the pool of 182 patients left aside for independent testing, we sampled,
with replacement, 120 participants’ data 20 times. We measured the performance
of PULSAR and the baseline models in each of these replicates to determine the
average and variation of the various metrics. The average accuracy, precision,
recall, F1 (macro as well as weighted) and AUROC are reported in Table 4.
While the performance of PULSAR degraded from validation to the independent
test set, it nevertheless outperformed all the other models in the independent
testing. And the overall comparative trend among the models also held –
performance improved when either PU learning or adaptive convolution was added
to the joint stream based model. When both PU and adaptive convolutions were
added, performance was even better. Finally, when bone, velocity and
acceleration streams were added to create the PULSAR model, the performance
exceeded all the other models. Additionally, the standard deviations of all
the performance metrics were less than 3%, which points to the stability of
PULSAR.

Figure 5: Box and whisker plot for accuracy, macro F1 and AUROC of PULSAR and
baseline models.

Figure 5 shows the box and whisker plot for accuracy, macro F1 and AUROC of
the different models. The addition of PU learning to the JS model made the
results less dispersed. It thus added stability to the model performance. The
addition of adaptive convolution, on the other hand, made the distribution of
accuracy normal like. The median line for PULSAR is above the box for each
metric of the baseline models. This is an indication that likely PULSAR is
statistically significantly different than the other models. This is more
rigorously tested in the next section.

### Statistical Significance Test

We have conducted Friedman non-parametric statistical test (FMT) (Demšar 2006)
on the independent test accuracy of the predictors, with the significance
level, $\alpha=0.05$. The Friedman statistic distributed according to Chi-
square with 4 degrees of freedom ($df$) was 68.16 ($p$-value 3.58e-11). As
this is larger than the critical value (9.488 for $df=4$), the null hypothesis
($H_{0}$: All the models have identical accuracy) is rejected. Table in Figure
6 summarizes the average rank of the predictors, where PULSAR comes out on
top. Subsequently, we conducted Post hoc Holm test for pairwise comparisons
for $\alpha=0.05$. The adjusted $p$-values are shown as a heatmap in Figure 6.
Based on these adjusted $p$-values and the results in Table 4, PULSAR’s
superiority over the JS, JS_PU and JS_AC models is statistically significant.
PULSAR is not statistically significantly different than the JS_AC_PU model,
though it has a better average accuracy over the latter.

Figure 6: Heatmap of the adjusted $p$-values from the post hoc Holm test on
the accuracy metric. Average rank of models based on the Friedman test is
shown in the table.

## Discussion

PULSAR employs adaptive graph convolutional neural network for PD screening
from the videos of the finger-tapping task. For graph construction, we have
used our knowledge of human anatomy and constrained it with geometry. PULSAR’s
performance demonstrate the potential of leveraging advanced deep learning
techniques to facilitate PD screening. Rizzo et al. showed that the accuracy
of clinical diagnosis of PD is 73.8%, when performed mainly by non-experts in
movement disorders (i.e., general neurologists, geriatricians, or general
practitioners). In our experiments, PULSAR achieved 80.95% accuracy in
validation set and a mean accuracy of 71.29% (2.49% standard deviation) in the
independent test set. This shows the potential of PULSAR to be integrated into
the health care system, provided that due consideration has been given to data
security, privacy and ethics.

Early diagnosis of PD followed by regular physician visits is crucial to
improve the quality of life as well as life expectancy of a PD patient (Fujita
et al. 2021). As the projected total economic burden in US due to PD surpasses
$79 billion by 2037 (Yang et al. 2020), early PD diagnosis is not only crucial
for the patients themselves, but also for the sustainability of the overall
healthcare system. However, early PD symptoms can be mistaken for those of
normal aging. Therefore, an undiagnosed PD patient may not make the first
visit to a neurologist for a long time. This is specially so in healthcare
settings with an extreme scarcity of neurologists. PULSAR can have a
significant impact in public health by making the screening easily accessible
to anyone anywhere with webcam and computer to record the fingar-tapping task,
and internet connection to send to video over. The AI model can then
automatically assess the symptoms and refer the patient to a neurologist if
necessary.

To train PULSAR, we augmented the dataset by flipping the videos. It is also
worthwhile to introduce various types of noise, such as, blurriness, and
heterogeneous lighting. These can occur naturally in home setting, and the
model needs to be robust against such variations in videos. Also, PD symptoms
can be suppressed for patients on medication, which can confuse the model.
When medication related information is available, the training should
incorporate that to improve the model’s learning. This can be an interesting
future direction. In future, we plan to improve PULSAR’s performance using a
larger training set. In addition, we envision extending our proposed technique
to assess other movement disorders like ataxia and Huntington’s disease. Our
work shows promise in advancing PD screening accessibility and warrants
further research to improve its robustness and scope. However, note that
PULSAR is a support tool, not a substitute for clinical evaluation.

## References

  * Demšar (2006) Demšar, J. 2006.  Statistical comparisons of classifiers over multiple data sets.  _The Journal of Machine learning research_ , 7: 1–30.
  * Du Plessis, Niu, and Sugiyama (2014) Du Plessis, M. C.; Niu, G.; and Sugiyama, M. 2014.  Analysis of learning from positive and unlabeled data.  _Advances in neural information processing systems_ , 27.
  * Elkan and Noto (2008) Elkan, C.; and Noto, K. 2008.  Learning classifiers from only positive and unlabeled data.  In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 213–220.
  * Fujita et al. (2021) Fujita, T.; Babazono, A.; Kim, S.; Jamal, A.; and Li, Y. 2021.  Effects of physician visit frequency for Parkinson’s disease treatment on mortality, hospitalization, and costs: a retrospective cohort study.  _BMC geriatrics_ , 21(1): 1–12.
  * Govindu and Palwe (2023) Govindu, A.; and Palwe, S. 2023.  Early detection of Parkinson’s disease using machine learning.  _Procedia Computer Science_ , 218: 249–261.
  * Hughes et al. (1992) Hughes, A. J.; Daniel, S. E.; Kilford, L.; and Lees, A. J. 1992.  Accuracy of clinical diagnosis of idiopathic Parkinson’s disease: a clinico-pathological study of 100 cases.  _Journal of neurology, neurosurgery & psychiatry_, 55(3): 181–184.
  * Islam et al. (2023) Islam, M. S.; Rahman, W.; Abdelkader, A.; Lee, S.; Yang, P. T.; Purks, J. L.; Adams, J. L.; Schneider, R. B.; Dorsey, E.; and Hoque, E. 2023.  Using AI to Measure Parkinson’s Disease Severity at Home.  _npj Digital Medicine_.
  * Ji et al. (2018) Ji, W.; Lv, Y.; Lin, G.; and Li, Y. 2018.  Filtering-based domain adaptation model fusion method in prediction of Parkinson’s disease symptom severity.  _Chinese Journal of Scientific Instrument_ , 39(6): 104–111.
  * Johri, Tripathi et al. (2019) Johri, A.; Tripathi, A.; et al. 2019.  Parkinson disease detection using deep neural networks.  In _2019 twelfth international conference on contemporary computing (IC3)_ , 1–4. IEEE.
  * Keskar et al. (2016) Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2016\.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_.
  * Kim et al. (2021) Kim, M.; Kim, J.; Qu, J.; Huang, H.; Long, Q.; Sohn, K.-A.; Kim, D.; and Shen, L. 2021.  Interpretable temporal graph neural network for prognostic prediction of Alzheimer’s disease using longitudinal neuroimaging data.  In _2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 1381–1384. IEEE.
  * Kissani et al. (2022) Kissani, N.; Liqali, L.; Hakimi, K.; Mugumbate, J.; Daniel, G. M.; Ibrahim, E. A. A.; Yewnetu, E.; Belo, M.; Wilmshurst, J.; Mbelesso, P.; et al. 2022.  Why does Africa have the lowest number of Neurologists and how to cover the Gap?  _Journal of the Neurological Sciences_ , 434: 120119.
  * Langevin et al. (2019) Langevin, R.; Ali, M. R.; Sen, T.; Snyder, C.; Myers, T.; Dorsey, E. R.; and Hoque, M. E. 2019.  The PARK framework for automated analysis of Parkinson’s disease characteristics.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 3(2): 1–22.
  * Li and Li (2022) Li, A.; and Li, C. 2022.  Detecting parkinson’s disease through gait measures using machine learning.  _Diagnostics_ , 12(10): 2404.
  * Li et al. (2019) Li, Y.; He, Z.; Ye, X.; He, Z.; and Han, K. 2019.  Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition.  _EURASIP Journal on Image and Video Processing_ , 2019(1): 1–7.
  * Marek et al. (1996) Marek, K.; Seibyl, J.; Zoghbi, S.; Zea-Ponce, Y.; Baldwin, R.; Fussell, B.; Charney, D.; Van Dyck, C.; Hoffer, P.; and Innis, R. 1996.  [sup 123 I] beta-CIT/SPECT imaging demonstrates bilateral loss of dopamine transporters in hemi-Parkinson’s disease.  _Neurology_ , 46(1): 231–237.
  * Nguyen, Li, and Ng (2011) Nguyen, M. N.; Li, X.-L.; and Ng, S.-K. 2011.  Positive unlabeled learning for time series classification.  In _Twenty-Second International Joint Conference on Artificial Intelligence_. Citeseer.
  * Qiu, Zhang, and Zhu (2009) Qiu, Q.; Zhang, Y.; and Zhu, J. 2009.  Building a text classifier by a keyword and unlabeled documents.  In _Advances in Knowledge Discovery and Data Mining: 13th Pacific-Asia Conference, PAKDD 2009 Bangkok, Thailand, April 27-30, 2009 Proceedings 13_ , 564–571. Springer.
  * Rizzo et al. (2016) Rizzo, G.; Copetti, M.; Arcuti, S.; Martino, D.; Fontana, A.; and Logroscino, G. 2016.  Accuracy of clinical diagnosis of Parkinson disease: a systematic review and meta-analysis.  _Neurology_ , 86(6): 566–576.
  * Senturk (2020) Senturk, Z. K. 2020.  Early diagnosis of Parkinson’s disease using machine learning algorithms.  _Medical hypotheses_ , 138: 109603.
  * Shahid and Singh (2020) Shahid, A. H.; and Singh, M. P. 2020.  A deep learning approach for prediction of Parkinson’s disease progression.  _Biomedical Engineering Letters_ , 10: 227–239.
  * Shi et al. (2019) Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019.  Two-stream adaptive graph convolutional networks for skeleton-based action recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 12026–12035.
  * Sivaranjini and Sujatha (2020) Sivaranjini, S.; and Sujatha, C. 2020.  Deep learning based diagnosis of Parkinson’s disease using convolutional neural network.  _Multimedia tools and applications_ , 79: 15467–15479.
  * Srinidhi, Ciga, and Martel (2021) Srinidhi, C. L.; Ciga, O.; and Martel, A. L. 2021.  Deep neural network models for computational histopathology: A survey.  _Medical Image Analysis_ , 67: 101813.
  * Wikipedia contributors (2022) Wikipedia contributors. 2022.  Parkinson’s disease — Wikipedia, The Free Encyclopedia.  [Online; accessed 12-August-2023].
  * Willis et al. (2022) Willis, A.; Roberts, E.; Beck, J.; Fiske, B.; Ross, W.; Savica, R.; Van Den Eeden, S.; Tanner, C.; Marras, C.; and M., P. F. P. G. A. R. S. M. R. B. C. H. C. T. W. B. D. J. 2022.  Incidence of Parkinson disease in North America.  _npj Parkinson’s Disease_ , 8(1): 170.
  * Yan, Xiong, and Lin (2018) Yan, S.; Xiong, Y.; and Lin, D. 2018.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Yang et al. (2020) Yang, W.; Hamilton, J. L.; Kopil, C.; Beck, J. C.; Tanner, C. M.; Albin, R. L.; Ray Dorsey, E.; Dahodwala, N.; Cintina, I.; Hogan, P.; et al. 2020.  Current and projected future economic burden of Parkinson’s disease in the US.  _npj Parkinson’s Disease_ , 6(1): 15.
  * Zhang et al. (2020) Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.-L.; and Grundmann, M. 2020.  Mediapipe hands: On-device real-time hand tracking.  _arXiv preprint arXiv:2006.10214_.
  * Zhao et al. (2022) Zhao, M.; Lei, H.; Huang, Z.; Zhang, Y.; Li, Z.; Liu, C.-M.; and Lei, B. 2022.  Attention-based Graph Neural Network for the Classification of Parkinson’s Disease.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , 4608–4614. IEEE.


# Model Barrier: A Compact Un-Transferable Isolation Domain
for Model Intellectual Property Protection

Lianyu Wang1111L. Wang and M. Wang contributed equally to this work., Meng
Wang2111L. Wang and M. Wang contributed equally to this work., Daoqiang
Zhang1222Corresponding author: D. Zhang<EMAIL_ADDRESS>and H. Fu
(hzfu@ieee.org)., Huazhu Fu2222Corresponding author: D. Zhang
<EMAIL_ADDRESS>and H. Fu (hzfu@ieee.org).
1College of Computer Science and Technology, Nanjing University of Aeronautics
and Astronautics, China.
2Institute of High Performance Computing (IHPC), Agency for Science,
Technology and Research (A*STAR), Singapore 138632.

###### Abstract

As scientific and technological advancements result from human intellectual
labor and computational costs, protecting model intellectual property (IP) has
become increasingly important to encourage model creators and owners. Model IP
protection involves preventing the use of well-trained models on unauthorized
domains. To address this issue, we propose a novel approach called Compact Un-
Transferable Isolation Domain (CUTI-domain), which acts as a barrier to block
illegal transfers from authorized to unauthorized domains. Specifically, CUTI-
domain blocks cross-domain transfers by highlighting the private style
features of the authorized domain, leading to recognition failure on
unauthorized domains with irrelevant private style features. Moreover, we
provide two solutions for using CUTI-domain depending on whether the
unauthorized domain is known or not: target-specified CUTI-domain and target-
free CUTI-domain. Our comprehensive experimental results on four digit
datasets, CIFAR10 & STL10, and VisDA-2017 dataset demonstrate that CUTI-domain
can be easily implemented as a plug-and-play module with different backbones,
providing an efficient solution for model IP protection.

##  1 Introduction

Figure 1: Model IP protection with our proposed CUTI-domain. Left: In standard
supervised learning (SL), the model owner trains a high-performance model on
the authorized domain (pink square) and then authorizes a specific user.
Authorized users have the right to use the model on the authorized domain to
get the correct prediction. However, a stealer can easily access the model on
the unauthorized domain (blue square), which violates the legitimate rights
and interests of the model owner. Right: Our method constructs a CUTI-domain
between the authorized and unauthorized domains, which could block the illegal
transferring and lead to a wrong prediction for unauthorized domains.

The recent success of deep learning models heavily relies on massive amounts
of high-quality data, specialized training resources, and elaborate manual
fine-tuning [55, 5, 56, 61]. Obtaining a well-trained deep model is both time-
consuming and labor-intensive [8]. Therefore, it should be protected as a kind
of scientific and technological achievement intellectual property (IP) [54,
9], thereby stimulating innovation enthusiasm in the community and further
promoting the development of deep learning. As shown in Fig. 1, in supervised
learning (SL), the model owner uses the overall features of the authorized
domain (pink square) for training, obtains a high-performance model, and
grants the right to use it to a specific user. Authorized users can use the
model on the authorized domain to obtain correct predictions. However, since
the model is trained with overall features, its potential feature
generalization region is large and may cover some unauthorized domains.
Therefore, there is a natural pathway between the authorized domain and the
unauthorized domain, and the released high-performance model obtained by SL
can be illegally transferred to the unauthorized domain (blue squares) through
methods such as domain adaptation [15, 16], and domain generalization [17,
18], to obtain correct prediction results. This presents a challenge in
protecting well-trained models. One of the most concerning threats raised is
“Will releasing the model make it easy for the main competitor to copy this
new feature and hurt owner differentiation in the market?” Thus, the model IP
protection has been proposed to defend against model stealing or unauthorized
use.

A comprehensive intellectual property (IP) protection strategy for deep
learning models should consider both ownership verification and applicability
authorization [10, 57]. Ownership verification involves verifying who has
permission to use the deep model by embedding watermarks [11, 12], model
fingerprint [13], and predefined triggers [14]. The model owner can grant
usage permission to a specific user, and any other users will be infringing on
the owner’s IP rights. However, an authorized user can easily transfer the
model to an unauthorized user, so the model owner must add special marks
during training to identify and verify ownership. Moreover, these methods are
vulnerable to fine-tuning, classifier retraining, elastic weight consolidation
algorithms, and watermark overwriting, which can weaken the model’s
protection. On the other hand, applicability authorization involves verifying
the model’s usage scenarios. Users with permission can apply the deep model
for the tasks specified by the model owner, and it is an infringement to use
it for unauthorized tasks [10]. However, users can easily transfer high-
performance models to other similar tasks to save costs, which is a common and
hidden infringement. Therefore, if the performance of the model can be limited
to the tasks specified by the owner and reduced on other similar tasks,
unauthorized users will lose confidence in stealing and re-authoring the
model. To achieve this, a non-transferable learning (NTL) method is proposed
[10], which uses an estimator with a characteristic kernel from Reproducing
Kernel Hilbert Spaces to approximate and increase the maximum mean difference
between two distributions on finite samples. However, the authors only
considered using limited samples to increase the mean distribution difference
of features between domains and ignored outliers. The convergence region of
NTL is not tight enough. Moreover, the calculation of the maximum mean
difference is class-independent, which reduces the model’s feature recognition
ability in the authorized domain to a certain extent.

To address the challenges outlined above, we first propose a novel approach
called the Compact Un-Transferable Isolation (CUTI) domain to prevent illegal
transferring of deep models from authorized to unauthorized domains. Our
approach considers the overall feature of each domain, consisting of two
components: shared features and private features. Shared features refer to
semantic features, while private features include stylistic cues such as
perspective, texture, saturation, brightness, background environment, and so
on. We emphasize the private features of the authorized domain and construct a
CUTI-domain as a model barrier with similar private style features. This
approach prevents illegal transfers to unauthorized domains with new private
style features, thereby leading to wrong predictions. Furthermore, we also
provide two CUTI-domain solutions for different scenarios. When the
unauthorized domain is known, we propose the target-specified CUTI-domain,
where the model is trained with a combination of authorized, CUTI, and
unauthorized domains. When the unauthorized domain is unknown, we use the
target-free CUTI-domain, which employs a generator to synthesize unauthorized
samples that replace the unauthorized domain in model training. At last, our
comprehensive experimental results on four digit datasets, CIFAR10 & STL10,
and VisDA-2017 demonstrate that our proposed CUTI-domain effectively reduces
the recognition ability on unauthorized domains while maintaining strong
recognition on authorized domains. Moreover, as a plug-and-play module, our
CUTI-domain can be easily implemented within different backbones and provide
efficient solutions.111https://github.com/LyWang12/CUTI-Domain.

##  2 Related Work

###  2.1 Model IP Protection

There are currently two main categories of methods for IP protection,
including ownership verification and applicability authorization. For
ownership verification, the most classic method is watermarking embedding
[19]. Kuribayashi _et al_. [20] proposed a quantifiable watermark embedding
method, which reduces the variation caused by embedding watermarks. Adi  _et
al_. [21] proposed a tracking mechanism in a black-box way. However, such
watermark embedding approaches have been proved to be susceptible to some
watermark removal and watermark overwriting methods. In our experiments, a
simple watermark is embedded into the model for ownership verification by
triggering mis-classification. Comprehensive experimental results demonstrate
that the proposed CUTI-domain is resistant to the common watermark removal
methods.

Applicability authorization is derived from usage authorization. Usage
authorization usually uses a preset private key to encrypt the whole/part of
the network. Only authorized users can obtain the private key and then use the
model. There are many advanced methods for usage authorization. For example,
Alam  _et al_. [24]. proposed an explicit locking mechanism for a lightweight
deep neural network, utilizing S-Boxes with good cryptographic properties to
lock each training parameter of a DNN model. Without knowledge of the
legitimate private key, unauthorized access can severely degrade the accuracy
of the model. Song  _et al_. [25] analyzed and calculated the critical weight
of the deep neural network model, and significantly reduced the time cost by
encrypting the critical weight to lock the deep neural network model against
unauthorized use. Wang  _et al_. [10] proposed data-based applicability
authorization named NTL, which preserves model performance on authorized data
while degrading model performance in other data domains. Compared to the
above, we construct a new class-dependent CUTI-domain with infinite samples
whose features are more similar with the source domain. By decreasing the
performance of the model on the CUTI-domain and the target domain, the
generalization bound of the model can be compacter, thereby constraining the
model performance within the authorized source domain.

###  2.2 Domain Transferring

In practice, domain gaps may arise due to different data collection scenarios
in different domains. Domain adaptation (DA) and domain generalization (DG)
are common solutions used to alleviate domain gaps [59, 58]. DA refers to
transferring a model from a labeled source domain to an unlabeled but relevant
target domain where the target domain’s data is accessible during the training
process [26]. DG differs from DA in that the target domain is inaccessible
during model training [27, 28]. DANN [29] s a classic DA method that
introduced a gradient inversion layer and a domain discriminator to confuse
the feature distributions of the two domains. Subsequently, CDAN [30] further
introduced categorical information entropy into the domain discriminator to
alleviate the class mismatch problem. For DG, Tobin  _et al_. [31] used domain
randomization to generate more training data from simulated environments for
generalization in real environments. Prakash  _et al_. [32] further considered
the structure of the scene when randomly placing objects for data generation,
enabling the neural network to learn how to utilize context when detecting
objects. Recently, some methods have been proposed and successfully applied to
cross-domain applications by seeking an intermediate state between the source
domain and the target domain, emphasizing the similarity between domains to
improve the model’s transferability [33, 34, 35, 36, 37]. In contrast, this
paper aims to seek an intermediate state to highlight the difference between
the authorized source domain and the unauthorized target domain, constraining
the feature transferability of the model and protecting the scientific and
technological achievements IP of the model owner.

##  3 Methodology

In this section, we first present our proposed CUTI-domain with the aim of
developing a solution that limits the performance of the model to authorized
source domains and reduces feature recognition capabilities on the
unauthorized target domain. Then, depending on whether the unauthorized target
domain is known or not, we provide two solutions, target-specified CUTI-domain
and target-free CUTI-domain, for protecting the model IP.

###  3.1 Compact Un-Transferable Isolation Domain

In the deep neural network model, the overall features extracted by the
feature extractor include two abstract components, i.e., semantic features,
and style features. Semantic features reflect the structural information of
samples and play a leading role in sample recognition; while style feature
refers to a series of weakly related clues, such as perspective, texture,
saturation, brightness, and background environment. For different domains with
the same task, semantic features are shared, while style features are private.
Most of the previous DA and DG works have been devoted to improving feature
transferability between domains, i.e., strengthening the focus of the model on
shared features while suppressing seemingly disturbing private style features.
However, to protect the intellectual property of the model, this paper aims to
limit the feature recognition ability of the model by highlighting private
style features of the source domain through style transfer, thus leading to
the failure of recognition on target domains that contain irrelative private
style.

Style transfer techniques suggest that styles are homogeneous and composed of
repeated structural motifs. Two images can be considered similar in style if
the features extracted by a trained classifier share the same statistics [38,
39]. First- and second-order statistics are often used as style features due
to their computational efficiency. These statistics refer to the mean and
variance of the extracted features. On the other hand, semantic features only
contain pure semantic information and exclude any style information. Similar
to [40], the semantic feature $f_{s}$ of the extracted feature $f$ can be
obtained by removing the style features, as follows:

$f_{s}=\frac{{f-{\mu(f)}}}{{{\sigma(f)}}},$ (1)

where $\mu(f)$ and $\sigma(f)$ denote the mean and variance of $f$.
Furthermore, style can be re-assign by $f_{s}\cdot\gamma+\beta$, where
$\gamma$ and $\beta$ are learned parameters. Afterward, Huang  _et al_. [38]
further explored adapting $f$ to arbitrarily given style by using style
features of another extracted feature instead of learned parameters.

Figure 2: Illustration of our proposed CUTI-domain generator. The output
feature of the source domain $f_{s}^{l}$ and the CUTI-domain $f_{i}^{l}$ in
the $i$-th feature extractor block are sent to the CUTI-domain generator, and
then the mean $\mu^{l}$ and variance $\sigma^{l}$ of $f_{s}^{l}$ is fused with
$f_{i}^{l}$, so that the private style features of the updated $f_{i}^{l}$ are
closer to the feature of source domain $f_{s}^{l}$ while the original
semantics features are maintained.

Drawing inspiration from these ideas, we propose a novel CUTI-domain design
that incorporates style features from the source domain. Specifically, we
randomly extract style features from the source domain and fuse them with the
overall features of the CUTI-domain, making the private style features of the
CUTI-domain more similar to those of the source domain. By doing so, we reduce
the model’s ability to recognize features on both the CUTI-domain and the
target domain, thereby implicitly blocking the pathway between the source and
target domains and limiting the model’s performance to the source domains.

Our CUTI-domain is generated by the CUTI-domain generator, which is a
lightweight, and plug-and-play module, as shown in Fig. 2. $f_{s}^{l}$ and
$f_{i}^{l}$ represent the deep features of the $i$-th feature extractor block
in the source domain and the CUTI-domain, respectively. First, $f_{s}^{l}$ and
$f_{i}^{l}$ are sent into CUTI-domain generator in parallel, and then the mean
$\mu^{l}$ and variance $\sigma^{l}$ of $f_{s}^{l}$ are calculated according to
the channel as private style features, followed by a $1\times 1$ convolution
layer $Conv$. Next, the $\mu^{l}$ and $\sigma^{l}$ are multiplied and added
channel-wisely by $f_{i}^{l}$ as:

$f_{i}^{l}\leftarrow(f_{i}^{l}\bigotimes Conv(\sigma^{l}))\bigoplus
Conv(\mu^{l}).$ (2)

As can be seen in Fig. 2, the private style features of updated $f_{i}^{l}$
are closer to those of $f_{s}^{l}$, while retaining its original semantic
features. Through continuous computation, CUTI-domain generators can construct
a labeled CUTI-domain containing a similar private style to the source domain.

###  3.2 Model IP Protection with CUTI-domain

####  3.2.1 Target-Specified CUTI-Domain

We first introduce how to utilize our CUTI-Domain with a given unauthorized
target domain. Fig. 3 illustrates the whole framework trained with our
proposed CUTI-domain, which consists of $L$ feature extractor blocks, $L$
CUTI-domain generators and a classifier. $x_{s}$, $x_{i}$ and $x_{t}$ denote
the data of the source domian, CUTI-domain and target domain respectively.
During training, when $epoch=2e$, i.e., epoch is equal to an even number,
$x_{s}$ and $x_{i}$ are fed into the feature extractor blocks in parallel,
followed by a CUTI-domain generator. The classifier at the end of the network
is used to predict the category of the sample, and the prediction results are
denoted by $p_{s}$ and $p_{i}$, respectively. When $epoch=2e+1$, i.e., epoch
is equal to an odd number, $x_{s}$ and $x_{t}$ are input into the network
parallelly without CUTI-domain generators, $p_{s}$ and $p_{t}$ denote their
predicted results.

Figure 3: Illustration of the framework trained with our proposed CUTI-domain.
The whole framework consists of $L$ feature extractor blocks, $L$ CUTI-domain
generators, and a classifier. When the epoch is equal to an even/odd number,
samples of the source domain $x_{s}$ and CUTI-domain $x_{i}$ / target domain
$x_{t}$ are fed into the feature extractor block in parallel. $p_{s}$, $p_{i}$
and $p_{t}$ denote their prediction results.

Based on the designed framework, an alternative loss function $\cal L$
according to the epoch is utilized:

$\small{\cal
L}=\begin{cases}KL(p_{s}||y_{s})-KL(p_{{\color[rgb]{1,0,0}i}}||y_{{\color[rgb]{1,0,0}i}}),&\text{if}\;epoch=2e,\\\
KL(p_{s}||y_{s})-KL(p_{{\color[rgb]{1,0,0}t}}||y_{{\color[rgb]{1,0,0}t}}),&\text{if}\;epoch=2e+1,\end{cases}$
(3)

where $KL(\cdot)$ stands for Kullback–Leibler divergence. By reducing the
$KL(\cdot)$ between prediction results and the labels on the source domain,
and expanding the $KL(\cdot)$ prediction results and the labels on the CUTI-
domain/target domain, the feature recognition ability of the model in the
source domain can be gradually improved, and the ability in the CUTI-
domain/target domain can be gradually reduced, thus effectively constrain the
performance of the model within authorized source domains for IP protection.
Finally, we summarize the strategy of our proposed target-specified CUTI-
domain in Algorithm 1.

During training, we initialize the CUTI-domain with the target domain training
set, and feed data of source domain training set $x_{s}$, CUTI-domain $x_{i}$,
and target domain training set $x_{t}$ into the network in parallel to train
the model as in Algorithm 1. At test time, model performance is evaluated on
the source domain test set and the target domain test set. In an ideal
situation, the model can maintain high sample recognition ability on the
authorized source domain, but achieve poor performance on the unauthorized
target domain.

Algorithm 1 Target-Specified CUTI-Domain.

0: The source domain $x_{s}$, CUTI-Domain $x_{i}$, the target domain $x_{t}$,
number of feature blocks $L$, the model parameters $\theta$.

1: Initialize CUTI-domain with the target domain.

2: For $epoch=1$ to $Max_{epochs}$ do

3: If $epoch=2i+1$ do

4: Calculate the output of $x_{s}$, $x_{i}$ in the $l$-th feature block:
$f_{s}^{l}$, $f_{i}^{l}$.

5: For $l=1$ to $L$ do

6: Update $f_{i}^{l}$ according to Eq. (2).

7: End For

8: If $epoch=2i$ do

9: Calculate the output of $x_{s}$, $x_{t}$ in the $l$-th feature block:
$f_{s}^{l}$, $f_{t}^{l}$.

10: Update model parameters $\theta$ by Eq. (3).

11: End For

12: Return model parameters $\theta$.

####  3.2.2 Target-Free CUTI-Domain

Sometimes, the unauthorized target domain is unknown. In this case, we cannot
directly feed the target domain and CUTI-domain into the network for model
training. To solve this, a synthesized target domain could be utilized. For
example, Wang  _et al_. [10] designed a GAN-based method by freezing
parameters to generate synthesized samples in different directions in place of
the target domain train set. Although their method is able to generate high-
quality synthesized samples, the direction of the generator is specified and
there are certain omissions. Huang  _et al_. [38] designed an adaptive
instance normalization (AdaIN) method based on GAN, which can generate
synthesized images with a specific style for a content image.

To take full advantage, we add Gaussian noise to AdaIN to obtain synthesized
samples with random styles. Finally, we mix the synthesized samples generated
by the above two methods (e.g., GAN and AdaIN) to replace the target domain
training set and initialize the CUTI-domain. The framework is consistent with
the target-specified CUTI-domain and the training process is detailed in the
Appendix. During testing, the model is evaluated on source domain test set and
other unknown domain with the same task. Our goal is not to design GANs to
generate high-quality synthesized images, but to evaluate the ability of CUTI-
domain to block illegal feature transfer in the context of synthesized images.
We still focus on reducing the feature recognition ability of the model on
unauthorized target domains and maintaining it on the authorized source
domains.

Table 1: The accuracy ($\%$) of target-specified CUTI-domain on digit
datasets. The left of ‘$\Rightarrow$’ represents the accuracy of the model
trained on the source domain dataset with SL, and the right of ‘$\Rightarrow$’
is the accuracy of CUTI-domain. CUTI source/target drop represent the average
degradation (relative degradation) of the proposed CUTI-domain relative to SL
on the source/target domains. NTL source/target drop is calculated from the
original paper. The bold numbers indicate the best performance.

Source/Target | MT | US | SN | MM |  CUTI Source Drop$\downarrow$ |  CUTI Target Drop$\uparrow$ |  NTL Source Drop$\downarrow$ |  NTL Target Drop$\uparrow$
---|---|---|---|---|---|---|---|---
MT | 99.2 $\Rightarrow$ 99.1 | 98.0 $\Rightarrow$ 6.7 | 38.2 $\Rightarrow$ 5.6 | 67.8 $\Rightarrow$ 8.7 | 0.10 (0.10%) | 61.00 (88.56%) | 1.00 (1.01%) | 46.57 (75.60%)
US | 92.6 $\Rightarrow$ 10.0 | 99.7 $\Rightarrow$ 99.6 | 25.5 $\Rightarrow$ 6.8 | 41.2 $\Rightarrow$ 8.4 | 0.10 (0.10%) | 44.70 (80.72%) | 1.00(1.00%) | 38.67 (75.55%)
SN | 66.7 $\Rightarrow$ 9.2 | 70.5 $\Rightarrow$ 6.7 | 91.2 $\Rightarrow$ 90.9 | 34.6 $\Rightarrow$ 10.9 | 0.30 (0.33%) | 48.33 (81.73%) | 1.10(1.23%) | 40.60 (77.25%)
MM | 98.4 $\Rightarrow$ 9.5 | 88.4 $\Rightarrow$ 6.8 | 46.3 $\Rightarrow$ 7.6 | 95.4 $\Rightarrow$ 95.4 | 0.00 (0.00%) | 69.73 (88.75%) | 2.10(2.30%) | 60.10 (76.95%)
Mean | / | / | / | / | 0.13 (0.13%) | 55.94 (84.94%) | 1.30 (1.39%) | 46.48 (76.34%)

Figure 4: The accuracy ($\%$) of SL, target-specified NTL and target-specified
CUTI-domain on CIFAR10, STL10 and VisDA-2017. The left of ‘$\rightarrow$’
represents the source domain and the right of ‘$\rightarrow$’ is the target
domain. The bars with different colors in each subgraph represent the accuracy
of the corresponding method in the source domain, the target domain, and the
degradation (relative degradation) of the model performance, respectively. The
data of NTL is obtained by reproducing its source code. Table 2: The accuracy
($\%$) of ownership verification by SL and CUTI-domain. FTAL [21], RTAL [21],
EWC [22], AU [22] and Overwriting are state-of-the-art watermark removal
methods. Avg drop represents the average degradation of the test dataset with
watermark patch relative to test dataset without a watermark patch. The bold
numbers indicate the best performance.

Source without Patch | Training Methods | Watermark Removal Approaches on CUTI | Avg Drop$\uparrow$
---|---|---|---
SL | CUTI | FTAL [21] | RTAL [21] | EWC [22] | AU [22] | Overwriting | CUTI | NTL
[Test w/o Watermark ($\%$)] | [Test w/o Watermark ($\%$)]
MT | 99.0 $/$ 99.3 | 11.3 $/$ 99.1 |  9.0 $/$ 100.0 |  9.4 $/$ 100.0 |  9.7 $/$ 100.0 |  9.0 $/$ 100.0 |  9.4 $/$ 96.2 | 89.9 | 88.4
US | 99.8 $/$ 99.8 |  7.7 $/$ 99.8 |  8.0 $/$ 100.0 |  8.7 $/$ 100.0 |  9.4 $/$ 100.0 |  9.4 $/$ 100.0 |  8.7 $/$ 98.6 | 90.9 | 85.7
SN | 91.3 $/$ 92.3 |  9.9 $/$ 92.1 |  9.4 $/$ 98.3 | 13.9 $/$ 97.6 | 10.8 $/$ 100.0 |  8.7 $/$ 100.0 | 10.4 $/$ 95.8 | 87.7 | 79.0
MM | 96.6 $/$ 96.0 | 16.8 $/$ 96.0 | 14.3 $/$ 95.4 | 24.0 $/$ 98.6 | 14.6 $/$ 100.0 | 14.6 $/$ 100.0 | 14.9 $/$ 95.8 | 81.5 | 77.3
CIFAR | 83.3 $/$ 75.1 | 10.7 $/$ 86.8 | 14.9 $/$ 97.9 | 14.9 $/$ 93.8 | 14.9 $/$ 100.0 |  9.4 $/$ 97.2 | 16.7 $/$ 90.3 | 81.7 | 74.6
STL | 87.9 $/$ 93.2 | 22.0 $/$ 88.2 | 20.0 $/$ 96.9 | 26.4 $/$ 93.8 | 13.9 $/$ 100.0 | 22.9 $/$ 94.1 | 21.2 $/$ 89.6 | 74.0 | 74.0
VisDA | 93.6 $/$ 92.2 | 13.1 $/$ 94.1 | 15.0 $/$ 95.5 | 20.5 $/$ 95.1 | 15.3 $/$ 100.0 | 21.9 $/$ 95.1 | 19.4 $/$ 96.2 | 78.0 | 76.8
Mean | / | / | / | / | / | / | / | 83.4 | 79.4

Table 3: The accuracy ($\%$) of target-free CUTI-domain on digit datasets. The
left of ‘$\Rightarrow$’ represents the accuracy of the model trained on the
source domain dataset with SL, and the right of ‘$\Rightarrow$’ is the
accuracy of CUTI-domain. CUTI source/target drop represent the average
degradation (relative degradation) of the proposed CUTI-domain relative to SL
on the source/target domains. The data of NTL is obtained by reproducing its
source code. The bold numbers indicate the best performance.

Source/Target | MT | US | SN | MM |  CUTI Source Drop$\downarrow$ |  CUTI Target Drop$\uparrow$ |  NTL Source Drop$\downarrow$ |  NTL Target Drop$\uparrow$
---|---|---|---|---|---|---|---|---
MT | 99.2 $\Rightarrow$ 98.8 | 98.0 $\Rightarrow$ 6.7 | 38.2 $\Rightarrow$ 6.7 | 67.8 $\Rightarrow$ 13.1 | 0.40 (0.40%) | 59.17 (85.43%) | 0.70 (0.71%) | 57.30 (84.06%)
US | 92.6 $\Rightarrow$ 9.1 | 99.7 $\Rightarrow$ 99.7 | 99.1 $\Rightarrow$ 25.5 |  6.8 $\Rightarrow$ 8.5 | 0.60 (0.60%) | 44.97 (80.96%) | 0.60 (0.60%) | 42.90 (74.71%)
SN | 66.7 $\Rightarrow$ 11.9 | 70.5 $\Rightarrow$ 14.3 | 91.2 $\Rightarrow$ 88.7 | 34.6 $\Rightarrow$ 13.6 | 2.50 (2.74%) | 44.00 (74.19%) | 3.20 (3.51%) | 41.53 (64.09%)
MM | 98.4 $\Rightarrow$ 19.6 | 88.4 $\Rightarrow$ 6.8 | 46.3 $\Rightarrow$ 9.5 | 95.4 $\Rightarrow$ 95.1 | 0.30 (0.31%) | 65.73 (83.96%) | 2.00 (2.10%) | 63.03 (77.62%)
Mean | / | / | / | / | 0.95 (1.02%) | 53.47 (81.13%) | 1.63 (1.73%) | 51.19 (75.12%)

##  4 Experiment

###  4.1 Implementation Details

We evaluate our method on seven popular DA/DG benchmarks, i.e., MNIST (MT)
[41], USPS (US) [42], SVHN (SN) [43] and MNIST-M (MM) [44] are commonly used
digit datasets, containing ten digits from 0 to 9 extracted from vary scenes;
CIFAR10 [45] and STL10 [46] are all ten-class classification datasets. We
follow the procedure of French et al. [47] to process the dataset such that
the correspondence between them holds. VisDA-2017 [48] is a Synthetic-to-Real
dataset containing training (T) and validation (V) sets from 12 categories.
Following the general setup, we adopt accuracy ($\%$) as the performance
metric of each task.

For tasks with different complexity, different backbones are adopted to
compare with the method proposed by Wang (NTL)  _et al_. [10]. We used VGG-11
[49] for digit datasets, VGG-13 [49] for CIFAR10 and STL10, and VGG-19 [49]
for VisDA. Since VGG contains five feature extractor blocks, $L$ is set to 5
in this paper, and CUTI-domain generator is deployed after each max-pooling
layer in the block. Pre-trained models are used for all backbones for a fair
comparison. The implementation of our comprehensive experiments is based on
the public platform Pytorch and an NVIDIA GeForce RTX 3090 GPU with 24GB of
memory. The batch size for each domain is set to 32.

Figure 5: The accuracy ($\%$) of SL, target-free NTL and target-free CUTI-
domain on CIFAR10, STL10, and VisDA-2017. The left of ‘$\rightarrow$’
represents the source domain and the right of ‘$\rightarrow$’ is the target
domain. The bars with different colors in each subgraph represent the accuracy
of the corresponding method in the source domain, the target domain, and the
degradation (relative degradation) of the model performance, respectively. The
data of NTL is obtained by reproducing its source code. Table 4: The accuracy
($\%$) of applicability authorization on digit datasets, CIFAR10, STL10 and
VisDA-2017. CUTI authorized/other domain represent the average accuracy of
CUTI-domain on the authorized/other domain, CUTI drop denote the degradation
(relative degradation). The results of NTL are reproduced by its source code.
The bold numbers indicate the best performance.

Source with Path | Test with Path(%) | Test without Path(%) |  CUTI Authorized Domain$\uparrow$ |  CUTI Other Domain$\downarrow$ |  CUTI Drop $\uparrow$ |  NTL Authorized Domain$\uparrow$ |  NTL Other Domain$\downarrow$ |  NTL Drop $\uparrow$
---|---|---|---|---|---|---|---|---
MT | US | SN | MM | MT | US | SN | MM
MT | 100.0 | 14.3 | 17.6 | 12.9 | 10.3 | 8.6 | 18.3 | 14.1 | 100.0 | 13.7 | 86.27(86.27%) | 99.8 | 14.5 | 85.31(85.49%)
US | 9.6 | 99.2 | 14.9 | 10.7 | 10.2 | 6.7 | 8.6 | 10.6 | 99.2 | 10.2 | 89.01(89.73%) | 98.5 | 13.3 | 85.20(86.50%)
SN | 10.8 | 13.5 | 99.1 | 23.0 | 9.6 | 9.2 | 17.7 | 8.9 | 99.1 | 13.2 | 85.86(86.64%) | 99.3 | 15.8 | 83.51(84.10%)
MM | 8.9 | 9.1 | 15.0 | 99.5 | 10.0 | 9.1 | 11.9 | 25.9 | 99.5 | 12.8 | 86.66(87.09%) | 99.5 | 14.0 | 85.49(85.92%)
| CIFAR | STL | CIFAR | STL | /
CIFAR | 97.9 | 42.5 | 13.1 | 11.6 | 97.9 | 22.4 | 75.50(77.12%) | 97.5 | 24.6 | 72.90(74.77%)
STL | 29.0 | 99.9 | 13.3 | 15.2 | 99.9 | 19.2 | 80.73(80.81%) | 98.6 | 20.5 | 78.10(79.21%)
| T | V | T | V | /
T | 100.0 | 22.9 | 20.3 | 10.1 | 100.0 | 17.8 | 82.23(82.23%) | 100.0 | 23.3 | 76.70(76.70%)
Mean | / | 99.4 | 15.5 | 83.75(84.27%) | 99.0 | 18.0 | 81.03(81.81%)

###  4.2 Result of Target-Specified CUTI-Domain

We selected one as the source domain and one as the target domain from the
digital datasets of four different domains, and constructed 16 transfer tasks,
as shown in Table 1. The left of $\Rightarrow$ represents the accuracy of the
model trained on the source domain dataset using supervised learning (SL), and
the right of $\Rightarrow$ is the accuracy of CUTI-domain. CUTI source/target
drops represent the drop (relative drop) of the proposed CUTI-domain in the
source and target domains, respectively. As can be seen, the average drop of
CUTI-domain on the target and source domains is 55.94 (84.94%) and 0.13
(0.13%), respectively. The last two columns represent the average performance
degradation of NTL with values of 46.48 (76.34%) and 1.30 (1.39%),
respectively. Compared with NTL, the decline of CUTI-domain in the target
domain is higher, and the negative impact on the source domain is smaller. It
can be inferred that CUTI-domain can better reduce the sample recognition
ability of the model for the target domain, and the decreases on the source
domain are slight.

Fig. 4 shows the results on CIFAR10 $\rightarrow$ STL10, STL10 $\rightarrow$
CIFAR10 and T $\rightarrow$ V. The bars with different colors in each subgraph
represent the accuracy of the corresponding method in the source domain, the
accuracy in the target domain, and the degradation (relative degradation) of
the model performance, respectively. Where the results of NTL are reproduced
by its source code to obtain comparable experimental data. SL has the largest
generalization region, resulting in the highest classification accuracy on the
target domain. By blocking the pathway with model locker, we observe
successful target domain reduction in accuracy for NTL and CUTI-domain, with
higher degradation than SL. Meanwhile, regardless of the task, the degradation
of CUTI-domain is higher than that of NTL, indicating that CUTI-domain can
better compress the generalization region of the model.

###  4.3 Result of Ownership Verification

In this section, we conduct ownership verification of model by triggering
classification errors. Specifically, we add a regular backdoor-based model
watermark patch on the authorized source domain dataset followed by NTL [10],
and treat the processed source domain as the new unauthorized target domain.
The model classification accuracy of SL and CUTI-domain on the authorized
source domain without watermark patch and the unauthorized target domain with
watermark patch are shown in Table 2. As shown in the second and third columns
in the table, for SL, there is little difference in the accuracy before and
after embedding the watermark patch, so the model is not sensitive to the
watermark patch. While for CUTI-domain, after watermark patch embedding, the
accuracy on unauthorized target domain is greatly reduced, and this difference
in performance can be used to verify the ownership of the model. In addition,
we also test the robustness of CUTI-domain using FTAL [21], RTAL [21], EWC
[22], AU [22] and watermark overwriting, which are state-of-the-art model
watermark removal methods. For a fair comparison, the settings of these
watermark removal methods are all consistent with NTL. The last two columns of
Table 2 are the drop in accuracy for watermarked versus un-watermarked data.
It can be observed that both CUTI-domain and NTL can effectively resist the
attack of watermark removal method, and the performance of CUTI-domain is
about 4% higher.

###  4.4 Result of Target-free CUTI-Domain

As detailed in Section 3.3.2, when the target domain is unknown, we use
synthesized samples to replace the target domain training set, and use other
unknown domain datasets as the target domain test set, as shown in Table 3 and
Fig. 5. For a fair comparison, the data of NTL is obtained by reproducing its
source code. It can be observed that the average drop of CUTI-domain on the
target domain is higher than that of NTL, and the drop on the source domain is
lower, indicating better IP protection ability of our proposed CUTI-domain on
unauthorized domains.

Fig. 5 shows the results for CIFAR10 $\rightarrow$ STL10, STL10 $\rightarrow$
CIFAR10 and T $\rightarrow$ V. Where the results of NTL are reproduced by its
source code to obtain comparable experimental data. Consistent with the
previous text, the degradation of CUTI-domain and NTL are higher than SL, and
CUTI-domain is the highest, which implies that CUTI-domain can better compress
the generalization region of the model. Meanwhile, considering the complexity
of the VisDA-2017 dataset, it is more difficult to extract representative
features to build CUTI-domain, so the drop of T $\rightarrow$ V of CUTI-domain
is only slightly higher than NTL.

###  4.5 Result of Applicability Authorization

In this section, we validate the applicability of model by restricting its
generalization ability to the authorized source domain. Specifically, similar
to section 4.3, we add an authorized watermark patch on the source domain as a
new authorized source domain training set. Then the original source domain,
synthesized samples, and synthesized samples with authorized watermark patches
are mixed as the unauthorized target domain training set. During testing,
other unknown domains are used as the test set. The experimental results of
CUTI-domain are shown in Table 4, where the results of NTL are reproduced by
its source code. It can be observed that the model performs better on the
source domain with authorized watermark patch, but performs poorly on other
unknown domains with or without watermark patch. This is consistent with our
expectation that the generalization ability of the model is restricted to the
source domain with the authorized watermark patch. Meanwhile, the average drop
rate of our proposed CUTI-domain is 83.75 (84.27%), which is higher than NTL
with 81.03 (81.81%). This is mainly because NTL directly utilizes the limited
features of the source and target domains for distance maximization, while we
construct a CUTI-domain with infinite samples similar to the source domain,
whose generalization boundary is more compact, and thus the model IP
protection ability is stronger.

###  4.6 Ablation Study

Backbone: In this section, we verify the ability of IP protection of CUTI-
domain combined with other backbones on the VisDA-2017 dataset. As shown in
the left of Fig. 6, compared with SL, CUTI-domain can further reduce the
recognition ability on the target domain when implemented with VGG-19 [49],
ResNet-34 [50], Inceptionv3 [51], Xception [52] and SWIM [53], Meanwhile, the
accuracy of the CUTI-domain on target domain is lower when combined with
Xception [52] and SWIM [53], since they have stronger feature extraction
ability on complex datasets than VGG-19 [49], ResNet-34 [50] and Inceptionv3
[51], lead to building a CUTI-domain that is more similar to the source
domain, thus better compacting the model performance within the source domain,
implying stronger model IP protection ability.

Loss Function: We further explore the contribution of various parts of our
proposed alternative loss function $\cal L$ in Eq. (3). The variants of $\cal
L$ are designed as:

$\displaystyle{\cal L}_{1}$ $\displaystyle=KL(p_{s}||y_{s})-KL(p_{t}||y_{t}),$
(4) $\displaystyle{\cal L}_{2}$
$\displaystyle=KL(p_{s}||y_{s})-KL(p_{i}||y_{i}),$ (5) $\displaystyle{\cal
L}_{3}$ $\displaystyle=KL(p_{s}||y_{s})-KL(p_{t}||y_{t})-KL(p_{i}||y_{i}).$
(6)

We validate the performance of different loss function variants on three
random tasks, as shown in the right of Fig. 6. Due to the different complexity
of the datasets, the difficulties of feature extraction and CUTI-domain
construction are varying. On datasets with simple features (i.e., MT
$\rightarrow$ SN), $\cal L$ performs better, while on slightly more complex
datasets, $\cal L$ only slightly outperforms other loss functions. It can be
seen that the accuracy scores of different loss function variants on the
target domain are relatively close, and $\cal L$ has the lowest accuracy on
each dataset, implying the validity of our proposed alternative loss function
$\cal L$.

Figure 6: Left: The accuracy ($\%$) of SL and target-specified CUTI-domain
combined with different backbones on VisDA-2017. Right: The accuracy ($\%$) of
SL and target-specified CUTI-domain with different loss functions on three
random tasks.

##  5 Conclusion

In the field of artificial intelligence, protecting well-trained models as a
form of IP poses challenges. To address this issue, we propose a novel CUTI-
domain that acts as a barrier to constrain model performance to authorized
domains. Our approach involves creating an isolation domain with features
similar to those in the authorized domain, effectively blocking the model’s
pathway between authorized and unauthorized domains and leading to recognition
failure on unauthorized domains with new private style features. We also offer
two versions of the CUTI-domain, e.g., target-specified and target-free,
depending on whether the unauthorized domain is known. Our experimental
results on seven popular cross-domain datasets demonstrate the efficacy of our
lightweight, plug-and-play CUTI-domain module. We hope that our work could
promote the research of model IP protection and security, which should be
taken seriously in real-world applications.

Acknowledge: This work was supported by the National Natural Science
Foundation of China (Nos. 62136004, 62276130), the Key Research and
Development Plan of Jiangsu Province (No. BE2022842), and Huazhu Fu’s A*STAR
Central Research Fund and Career Development Fund (C222812010).

## References

  * [1] R. Waseem and Z. Wang, “Deep convolutional neural networks for image classification: A comprehensive review,” _Neural computation_ , pp. 2352–2449, 2017.
  * [2] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _CVPR_ , 2017, pp. 4700–4708.
  * [3] S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopo, “Image segmentation using deep learning: A survey,” _IEEE TPAMI_ , vol. 44, no. 7, pp. 3523–3542, 2021.
  * [4] S. Qi, X. Ning, G. Yang, L. Zhang, P. Long, W. Cai, and W. Li, “Review of multi-view 3d object recognition methods based on deep learning,” _Displays_ , vol. 69, p. 102053, 2021.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009, pp. 248–255.
  * [6] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _arXiv preprint arXiv:1611.01578_ , 2016.
  * [7] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for facial expression recognition,” in _ICCV_ , 2015, pp. 2983–2991.
  * [8] K. He, X. Zhang, n. Shaoqing Re, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [9] A. Chakraborty, A. Mondai, and A. Srivastava, “Hardware-assisted intellectual property protection of deep learning models,” in _57th ACM/IEEE Design Automation Conference_ , 2020, pp. 1–6.
  * [10] L. Wang, S. Xu, R. Xu, X. Wang, and Q. Zhu, “Non-transferable learning: A new approach for model ownership verification and applicability authorization,” in _ICLR_ , 2021.
  * [11] P. W. Wong and N. Memon, “Secret and public key image watermarking schemes for image authentication and ownership verification,” _IEEE Transactions on Image Processing_ , vol. 10, no. 10, pp. 1593–1601, 2001.
  * [12] C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 587–601.
  * [13] E. L. Merrer, P. Perez, and G. Tredan, “Adversarial frontier stitching for remote neural network watermarking,” _Neural Computing and Applications_ , vol. 32, no. 13, pp. 9233–9244, 2020.
  * [14] L. Fan, K. W. Ng, and C. S. Chan, “Rethinking deep neural network ownership verification: Embedding passports to defeat ambiguity attacks,” _Advances in Neural Information Processing Systems 32_ , 2019.
  * [15] W. Mei and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * [16] K. You, M. Long, Z. C. andJianmin Wang, and M. I. Jordan, “Universal domain adaptation,” in _CVPR_ , 2019, pp. 2720–2729.
  * [17] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Deep visual domain adaptation: A survey,” _IEEE TPAMI_ , pp. 1–20, 2022.
  * [18] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu, “Generalizing to unseen domains: A survey on domain generalization,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–1, 2022.
  * [19] C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in _Proceedings of the ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 587–601.
  * [20] M. Kuribayashi, T. Tanaka, and N. Funabiki, “Deepwatermark: Embedding watermark into dnn model,” in _Asia-Pacific Signal and Information Processing Association Annual Summit and Conference_ , 2020, pp. 1340–1346.
  * [21] Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in _27th USENIX Security Symposium_ , 2018, pp. 1615–1631.
  * [22] X. Chen, W. Wang, C. Bender, Y. Ding, R. Jia, B. Li, and D. Song, “Refit: a unified watermark removal framework for deep learning systems with limited data,” _arXiv preprint arXiv:1911.07205_ , 2019.
  * [23] Y. Nagai, Y. Uchida, S. Sakazawa, and S. Satoh, “Digital watermarking for deep neural networks,” _International Journal of Multimedia Information Retrieval_ , vol. 7, no. 1, pp. 3–16, 2018.
  * [24] M. Alam, S. Saha, D. Mukhopadhyay, and S. Kundu, “Deep-lock: Secure authorization for deep neural networks,” _arXiv preprint arXiv:2008.05966_ , 2020.
  * [25] Z. Song, W. Jiang, J. Zhan, X. Wen, and C. Bian, “Critical-weight based locking scheme for dnn ip protection in edge computing: work-in-progress,” in _Proceedings of the 2021 International Conference on Hardware/Software Codesign and System Synthesis_ , 2021.
  * [26] W. Gabriela, “Domain adaptation for visual applications: A comprehensive survey,” _arXiv preprint arXiv:1702.05374_ , 2017.
  * [27] S. Xu, L. Wang, Y. Wang, and Q. Zhu, “Weak adaptation learning: Addressing crossdomain data insufficiency with weak annotator,” in _ICCV_ , 2021, pp. 8917–8926.
  * [28] J. Dong, Y. Cong, G. Sun, Z. Fang, and Z. Ding, “Where and how to transfer: knowledge aggregation-induced transferability perception for unsupervised domain adaptation,” _IEEE TPAMI_ , pp. 1–1, 2021.
  * [29] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International Conference on Machine Learning_ , 2015, pp. 1180–1189.
  * [30] M. Long, Z. CAO, J. Wang, and M. I. Jordan, “Conditional adversarial domain adaptation,” _Advances in Neural Information Processing Systems 31_ , 2018\.
  * [31] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , 2017, pp. 23–30.
  * [32] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in _2019 International Conference on Robotics and Automation_ , 2019, pp. 7249–7255.
  * [33] B. Gong, Y. Shi, F. Sha, and K. Grauman, “Geodesic flow kernel for unsupervised domain adaptation,” in _CVPR_ , 2012, pp. 2066–2073.
  * [34] R. Gopalan, R. Li, and R. R. Chellappa, “Unsupervised adaptation across domain shifts by generating intermediate data representations,” _IEEE TPAMI_ , vol. 36, no. 11, 2013.
  * [35] Y. Dai, J. Liu, Y. Sun, Z. Tong, C. Zhang, and L.-Y. Duan, “Idm: An intermediate domain module for domain adaptive person re-id,” in _ICCV_ , 2021, pp. 11 864–11 874.
  * [36] Y. Sun, G. Yang, D. Ding, G. Cheng, J. Xu, and X. Li, “A gan-based domain adaptation method for glaucoma diagnosis,” in _International Joint Conference on Neural Networks_ , 2020, pp. 1–8.
  * [37] S. Li, M. Xie, K. Gong, C. H. Liu, Y. Wang, and W. Li, “Transferable semantic augmentation for domain adaptation,” in _CVPR_ , 2021, pp. 11 516–11 525.
  * [38] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _ICCV_ , 2017, pp. 1501–1510.
  * [39] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in _CVPR_ , 2017, pp. 6924–6932.
  * [40] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” _arXiv preprint arXiv:1610.07629_ , 2016.
  * [41] L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 141–142, 2012.
  * [42] J. J. Hull, “A database for handwritten text recognition research,” _IEEE TPAMI_ , vol. 16, no. 5, pp. 550–554, 1994.
  * [43] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” _NIPS Workshop_ , 2011.
  * [44] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [45] K. Alex and G. Hinton, “Learning multiple layers of features from tiny images,” _Technical Report_ , 2009.
  * [46] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , 2011, pp. 215–223.
  * [47] G. French, M. Mackiewicz, and M. Fisher, “Self-ensembling for domain adaptation,” _arXiv preprint arXiv:1706.05208_ , 2017.
  * [48] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko, “Visda: The visual domain adaptation challenge,” _arXiv preprint arXiv:1710.06924_ , 2017\.
  * [49] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [50] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [51] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _CVPR_ , 2016, pp. 2818–2826.
  * [52] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _CVPR_ , 2017, pp. 1800–1807.
  * [53] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, tephen Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021, pp. 9992–10 002.
  * [54] M. Xue, Y. Zhang, J. Wang, and W. Liu, “Intellectual Property Protection for Deep Learning Models: Taxonomy, Methods, Attacks, and Evaluations,” _IEEE Transactions on Artificial Intelligence_ , pp. 1–1, 2021.
  * [55] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, may 2015.
  * [56] R. Bommasani, D. A. Hudson _et al._ , “On the Opportunities and Risks of Foundation Models,” _arXiv_ , aug 2021. [Online]. Available: http://arxiv.org/abs/2108.07258
  * [57] R. Wang, J. Ren, B. Li, T. She, C. Lin, L. Fang, J. Chen, C. Shen, and L. Wang, “Free Fine-tuning: A Plug-and-Play Watermarking Scheme for Deep Neural Networks,” _arXiv_ , oct 2022. [Online]. Available: http://arxiv.org/abs/2210.07809
  * [58] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–20, 2022.
  * [59] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * [60] M. Ribeiro, G. Katarina, and M. A. Capretz, “Mlaas: Machine learning as a service,” in _IEEE 14th International Conference on Machine Learning and Applications (ICMLA)_ , 2015.
  * [61] F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in Medical Imaging: A Survey,” _arXiv_ , jan 2022. [Online]. Available: http://arxiv.org/abs/2201.09873